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Real  Time  Intelligent  Coaching  for  Command  and  Control 
Systems:  Phase  I  Final  Report 


Background 

This  report  contains  the  objectives  and  findings  of  Air  Force  Office  of  Scientific 
Research  (AFOSR)  contract  F49620-98-C-0048,  “STTR-I  Real  Time  Intelligent  Coaching  for 
Command  and  Control.”  This  work  was  performed  as  a  Small  Business  Technology  Transfer 
Research  (STTR)  effort  by  Micro  Analysis  &  Design,  Inc.  (MA&D),  the  University  of 
Pittsbtirgh,  and  Veridian  Corporation.  All  participants  for  this  effort  are  listed  in  Appendix  D. 

MA&D,  the  University  of  Pittsburgh,  and  Veridian  undertook  to  determine  the  feasibility 
of  using  Bayesian  network  technology  to  support  automated  performance  assessment  and 
intelligent  coaching  of  personnel  using  a  command  and  control  (C^)  workstation.  The  work  was 
designed  to  begin  addressing  the  a  perceived  need  to  develop  a  technology  that  will  supply,  to 
personnel  in  C^  task  situations,  performance  feedback  after  high  demand  events. 

In  today’s  world,  there  are  many  time-critical  decisions  to  be  made  by  personnel  in 
various  high  demand  occupations.  Such  occupations  include  air  traffic  control,  emergency 
response  work,  and  military  mission-critical  teams,  such  as  those  operating  in  Combat 
Information  Centers  (CICs)  on  naval  vessels  and  those  directing  aircraft  from  Airborne  Warning 
and  Control  System  (AWACS)  platforms.  Often  personnel  with  these  types  of  occupations  find 
the  environments  in  which  they  work  to  be  very  stressful,  due  to  the  need  for  quick  response,  the 
high  workload  level,  the  criticality  of  the  decisions  to  be  made  and  the  cost  of  errors.  All  of 
these  factors,  in  isolation  and  combined,  can  undermine  the  individual’s  or  team’s  ability  to 
make  correct  decisions  by  impacting  the  time  to  gather  and  thoroughly  process  information. 
Additionally,  there  may  be  a  lack  of  information  availability  or  a  degradation  of  information 
content,  which  can  have  negative  impact  on  information  utility.  Such  situations  can  lead  the 
personnel  involved  to  make  judgments  that  are  not  well-informed. 

Timely  and  specific  intelligent  feedback  or  coaching,  delivered  while  on  the  job  or  during 
training,  might  benefit  teams  or  individuals  by  ensuring  that  they  can  produce  decisions  based  on 
the  best  available  information.  This  type  of  intelligent  feedback  can  be  of  use  at  several  points 
o\  cr  the  span  of  the  operator’s  job.  First,  during  training,  such  feedback  could  be  used  directly 
by  the  operator  to  inform  him/herself  about  strengths  and  weaknesses  in  either  overt  behaviors  or 
eognitive  task  performance  or  both.  The  instructor  would  also  be  able  to  use  this  information  in 
the  design  of  follow-on  exercises  for  subsequent  training.  It  would  be  especially  useful  if  such 
intelligent  feedback  could  drive  an  exercise  authoring  eind  development  system  such  that  the 
feedback  would  “pre-select”  training  objectives  as  the  basis  of  the  next  exercise. 

During  task  performance,  intelligent  feedback  or  coaching  could  be  used  at  three  points. 
First,  the  coach  could  be  used  before  the  operator  begins  his/her  scheduled  tasks  to  provide  some 
initial  guidance  as  to  events  to  expect  and  actions  to  take  in  response  to  those  events.  During  the 
operator’s  shift,  the  intelligent  coach  could  provide  guidance  and  information  during  lulls  in 


1 


activity  Giving  feedback  during  lulls  in  the  action  would  allow  the  operator  time  to  reflect  on 
the  information,  and,  thus,  potentially  allow  him/her  to  have  better  access  to  it  during  times  of 
higher  workload.  Finally,  at  the  end  of  the  operator’s  shift,  the  intelligent  coach  would  supply 
the  operator  with  a  debrief  of  his/her  performance  during  the  shift,  with  strengths  and 
weaknesses  highlighted  and  specific  suggestions  for  ways  to  attain  better  performance  during 
subsequent  shifts.  This  debrief  would  focus  on  both  decision  outcomes  (e.g.,  number  of  kills  or 
safe  landings)  and  decision  processes  (e.g.,  level  of  situation  awareness  or  knowledge  of 
heuristics).  Carolan  and  his  colleagues  (1997)  explored  a  similar  approach  performed  for  the 
Navy. 

Additionally,  such  a  tool  would  have  great  utility  for  the  research  community  studying 
decision-making,  as  well.  It  would  serve  as  a  way  to  gather  and  process  data  during  research 
trials  or  during  longitudinal  studies  using  real-world  events.  The  focus  on  the  measurement 
process  variables  would  allow  researchers  to  gain  a  greater  understanding  of  various  aspects  of 
decision-making  activities. 

Finally,  to  be  of  greatest  overall  use,  such  an  intelligent  coach  should  be  developed  using 
methods  that  are  not  domain-specific.  Domain  independence  of  the  methods  would  ensure  their 
transportability  and  utility  to  fields  other  than  the  one  in  which  they  are  initially  developed. 
Domain  independence  should  be  supported  by  platform  independence,  as  well.  Thus,  the 
methods  embodied  in  an  intelligent  coach  should  be  usable  with  any  type  of  workstation, 
given  that  there  is  a  means  by  which  data  can  be  accessed  from  the  station  and  input  into  the 
intelligent  coach,  and  methods  whereby  feedback  can  be  delivered  to  the  trainee  by  the  coach. 

Additionally,  the  impact  of  the  intelligent  coach  upon  performance  must  be  quantifiable. 
Thus,  methods  for  determining  its  utility,  and  that  of  other  feedback  techniques,  within  complex, 
dynamic  environments  must  be  developed.  However,  given  the  fluidity  of  the  events  that  occur 
in  such  environments,  often  it  is  very  difficult  to  determine  what  is  a  correct  operator  action, 
incorrect  action,  or  one  that  has  no  impact  until  after  the  current  action  has  occurred  and 
subsequent  events  and  actions  have  intervened.  This  means  that  the  coach  must  be  able  to 
predict  appropriate  performance  based  on  sparse  or  ambiguous  performance  data,  and  respond 
with  information  to  .the  user.  Given  the  sparseness  of  information  upon  which  a  coach  must  base 
feedback,  even  if  it  includes  an  expert  model  to  help  predict  performance,  it  may  not  make  sense 
to  down-grade  its  rated  performance  early  during  a  scenario  or  set  of  activities  to  which  the 
operator  must  respond. 

As  mentioned  earlier,  we  examined  the  feasibility  of  using  Bayesian  networks  as  the 
heart  of  our  performance  assessment  process  that  would  supply  information  to  an  intelligent 
coach  function.  Until  recently,  the  complexity  of  the  assessments  required  by  trainee  modeling 
made  Bayesian  reasoning  computationally  intractable.  However,  a  revolutionary  technology  has 
been  developed  for  doing  complex  Bayesian  reasoning  quickly.  It  is  based  on  Bayesian 
Networks  (also  called  Belief  Networks,  Causal  Networks  or  Graphical  Models).  Bayesian 
Networks  are  now  widely  used  in  AI  applications  such  as  medical  and  business  decision-making. 
Using  Bayesian  networks  for  trainee  modeling  is  a  rapidly  growing  area  of  research  (Conati,  et 
al.,  1997;  VanLehn,  1996),  and  it  is  at  a  level  of  maturity  that  it  can  begin  to  be  transitioned  into 
even  more  complex,  time-stressed,  high-demand  domains  such  as  those  represented  by  command 
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and  control  operations.  Thus,  we  planned  to  examine  structuring  an  intelligent  coach  around  this 
new,  but  maturing  technology. 

Bayesian  reasoning  is  most  useful  for  diagnostic  assessments,  such  as  are  needed  to  give 
intelligent  feedback  to  trainees  or  operators  within  dynamic  and  informationally  ambiguous 
domains,  such  as  the  environment.  An  assessment  is  diagnostic  when  the  goal  is  to  determine 
which  of  multiple  sub-skills,  factors  or  competencies  are  responsible  for  the  trainee's  or 
operator’s  performance.  To  put  it  in  cognitive  terms,  the  goal  of  a  diagnostic  assessment  is  to 
find  out  WHAT  the  trainee  or  operator  knows.  There  is  a  focus  on  the  process,  rather  than  the 
outcomes  of  the  task.  Diagnostic  assessments  are  typically  used  for  planning  feedback  or 
remediation.  Non-diagnostic  assessments,  which  are  often  used  for  placement  or  advancement, 
find  out  only  HOW  MUCH  the  trainee  knows  and  not  exactly  what  the  trainee  knows.  Bayesian 
reasoning  is  virtually  required  when  the  assessment  of  a  trainee  or  operator  is  underdetermined 
by  the  available  performance  data.  A  diagnostic  assessment  is  underdetermined  when  there  are 
many  different  ways  to  explain  the  trainee’s  or  operator’s  performance.  For  instance,  if  an 
operator  or  trainee  gets  a  problem  wrong  or  performs  a  task  incorrectly  and  there  are  10  different 
sub-skills  required  for  correct  performance,  then  it  is  not  clear  which  combination  of  the  sub¬ 
skills  is  missing.  The  data  underdetermine  the  assessment.  By  acquiring  more  performance 
data,  the  assessment  can  often  be  disambiguated.  For  instance,  having  the  trainee  answer  100 
problems,  or  ensure  that  an  operator  faces  similar  task  situations  100  times  might  suffice  to 
determine  which  of  the  210  possible  combinations  of  sub-skills  uniquely  characterize  this 
performance.  However,  with  complex  tasks,  such  as  the  ones  addressed  by  the  proposed 
research,  the  number  of  competencies/factors  to  be  assessed  is  quite  high  compared  to  the 
amount  of  performance  data  available.  With  such  complex  tasks,  assessments  are  almost  always 
underdetermined  by  the  data. 

When  an  assessment  is  underdetermined  by  the  data,  human  trainers  do  not  simply  give 
up,  but  use  their  experience  with  other  trainees  to  guess  the  most  likely  explanation  for  the 
trainee's  behavior.  In  a  training  situation,  it  is  better  to  give  feedback  and  remediation  based  on  a 
guess  than  no  feedback  at  all.  Bayesian  reasoning  is  simply  a  principled,  mechanical  means  of 
making  guesses  based  on  experience  when  the  data  underdetermine  the  diagnosis.  Bayesian 
reasoning  is  driven  by  PRIORS.  For  each  competency/sub-skill/factor,  a  Bayesian  reasoner  is 
gi\  en  a  probability,  called  the  prior  probability,  that  a  randomly  drawn  trainee  or  operator  from 
the  target  population  will  have  that  sub-skill/factor/competencies.  (Note:  This  assumes  the  prior 
probabilities  of  individual  competencies  are  independent;  More  complicated  representations  than 
priors  arc  necessary  otherwise.)  Priors  can  be  tabulated  from  assessment  of  a  set  of  trainees  or 
operators  obtained  by  using  human  assessors  or  some  other  "gold  standard."  There  are  also 
bootstrapping  techniques  to  calculate  priors  using  Bayesian  techniques.  Essentially,  the  priors 
represent  the  same  information  as  that  which  human  trainers  rely  upon  in  diagnosing  trainees 
when  the  data  are  ambiguous,  namely,  the  distinction  between  typical  and  rare  trainee  knowledge 
maladies.  The  relationship  of  the  prior  probabilities  to  the  resulting  probabilities  can  be  seen  in 
Figure  1 . 
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Figure  1.  Relationship  Between  Prior  Probabilities  and  Post-task  Probabilities 

The  trainee  modeling  component  of  a  typical  intelligent  tutoring  system  is  essentially  an 
assessment  of  the  trainee's  competence  as  well  as  their  plans,  goals  and  intentions  during  the 
performance.  It  is  an  extremely  underdetermined  diagnostic  assessment.  Bayesian  reasoning 
should  provide  much  more  accurate  diagnoses  than  the  heuristic  reasoning  that  is  typically  used. 
Thus,  higher  quality  feedback  should  be  able  to  be  supplied  to  the  trainee  or  operator. 

Thus,  in  Phase  I  of  this  effort,  our  goal  was  to  develop  a  proof-of-concept  intelligent 
coaching  capability  consisting  of  four  modules:  1)  the  expert  model  module  (which  for  the 
proof-of-concept  demonstration,  contains  the  steps  that  an  expert  AWACS  Weapons  Director 
team  would  take  with  regard  to  the  target  task  of  initially  committing  aircraft  to  pursue  enemy  air 
targets),  2)  the  comparison  module  that  accepts  operator/team  data  and  compares  it  to  output 
from  the  expert  model,  3)  the  assessment  module,  which  takes  output  from  the  comparison 
module  and  determines,  using  differences  in  linkage  weights  in  a  Bayesian  network,  the  possible 
locus  and  extent  of  performance  discrepancies  between  the  model  and  the  operator  performance, 
and  4)  the  feedback  module,  which  places  the  results  of  the  assessment  module  in  a  report, 
presenting  information  concerning  competency  levels.  To  meet  our  requirements  we  had  four 
technical  objectives: 

1 .  Develop  an  intelligent  coaching  method,  based  on  Bayesian  network  technology,  to 
supply  intelligent,  well-defined,  task-specific  feedback  to  users  and  teams  of  a 
workstation. 

2.  Integrate  intelligent  coach  output  with  a  command  and  control  workstation. 

3.  Begin  to  assess  the  impact  to  performance  of  the  developed  intelligent  coaching 
method. 
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4.  Investigate  the  potential  of  capturing  verbal  communications  for  inclusion  as 
information  sources  and  content  into  the  operator  model  contained  within  the 
intelligent  coach. 

The  first  objective  contains  the  primary  focus  of  the  effort,  the  development  of  an 
intelligent  coaching  method,  using  Bayesian  network  technolo^.  The  earlier  discussion 
presents  our  reasons  for  selecting  this  approach  for  generating  intelligent  feedback  to  operators. 

Our  second  objective  was  to  integrate,  at  some  level,  the  intelligent  coach  with  a 
command  and  control  workstation.  To  show  that  such  output  can  be  integrated  with  a 
workstation,  we  determined  that  at  the  present  time,  the  best  approach  to  integration  is  a  strap- 
on”  method  in  which  operator  data  would  be  gathered  from  a  workstation  and  supplied  to  a 
processor  that  would  generate  the  intelligent  feedback.  The  feedback  would  then  be  available  for 
review  by  the  operators.  This  approach  would  allow  the  operators  time  to  reflect  upon  the 
feedback  prior  to  needing  to  implement  it.  It  would  also  maintain  the  integrity  of  the  operational 
or  training  system. 

It  should  be  noted  that  we  elected  to  present  feedback  during  a  period  of  relatively  low 
workload  because  of  the  potential  negative  impact  that  true  real-time  feedback  can  have  on 
performance.  There  are  two  major  difficulties  with  giving  true  real-time  feedback  to  an  operator 
(real-time  feedback  referring  to  the  potential  of  feedback  occurring  after  any  or  every  action 
taken  by  an  operator  during  task  performance).  First,  we  examined  several  high-demand 
command  and  control  systems  in  which  time-critical  decisions  must  be  made  (for  example 
AEGIS  and  AW  ACS).  Our  examination  included  observations  of  performance  on  such  systems 
and  discussions  with  subject  matter  experts.  From  this  initial  examination,  we  determined  that 
interrupting  the  operator  while  he  or  she  is  performing  the  tasks  at  hand  in  order  to  supply 
feedback  may  result  in  a  detriment  in  performance  due  to  the  intrusive  nature  of  the  feedback. 
This  suggested  that  feedback  should  be  prepared  in  real  time,  but  its  presentation  saved  until  the 
operator  can  reflect  upon  it. 

Second,  in  order  to  give  feedback  to  an  operator  while  in  the  midst  of  performing  tasks 
during  a  critical  real-time  operation,  the  assessment  and  coaching  system  initially  must  predict 
the  operator’s  knowledge  state  based  on  extremely  limited  information.  In  a  non-training 
situation,  the  coach  must  also  predict  the  knowledge  states  and  potential  behaviors  of  others  in 
the  operation  whose  knowledge  states  and  behavior  patterns  may  be  unknowable,  for  example, 
pilots  in  hostile  aircraft.  During  the  early  portions  of  the  operation,  the  coach  may  not  have 
enough  information  as  to  the  state  of  events  and  availability  of  information  to  the  operator  to 
generate  useful  or  focused  feedback.  In  this  type  of  situation,  what  may  be  more  useful  to  the 
operator  is  a  workstation  interface  that  highlights  the  information  that  is  the  most  critical  for 
optimal  task  performance.  However,  this  is  a  re-designed  interface,  not  an  intelligent  coach.  A 
prototype  of  such  a  system  has  recently  been  developed  to  support  some  AEGIS  functions 
(Morrison,  et.  al.,  1997). 

Our  third  objective  addresses  the  need  to  begin  early  assessment  of  the  utility  of  the 
output  of  the  technology.  In  order  to  determine  whether  a  method  or  approach  should  be 
considered  for  further  study,  knowing  its  feasibility  is  not  enough.  The  method,  in  addition  to 
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being  feasible,  must  be  useful.  To  meet  our  third  objective,  we  intended  to  begin  developing 
strategies  for  determining  the  utility  of  our  method  for  using  Bayesian  networks  for  identifying 
areas  in  which  operators  need  feedback. 

Finally,  our  fourth  objective  was  to  examine  the  potential  for  capturing  and  including 
verbal  communications  into  the  performance  analysis.  Verbal  communications,  for  high- 
demand,  decision-making  teams,  is  often  an  important  method  by  which  information  is  conveyed 
to  other  team  members,  workload  is  re-distributed,  decision  processes  occur,  and  team 
cohesiveness  is  maintained.  Frequently,  capture  of  communication  initiation  and  reception  can 
occur  and  be  correlated  to  other  capturable  environmental  events  available  from  the 
workstations,  but  often  the  uncapturable  content  of  the  those  communications  reflect  details  of 
operator  knowledge  that  cannot  be  captured  in  any  other  way.  Therefore,  in  order  for  any 
coaching  or  feedback  mechanism  to  be  truly  reflective  of  the  operator’s  knowledge  state,  the 
content  of  verbal  communications  needs  to  be  factored  into  the  current  state  of  the  coach’s  model 
of  the  operator. 

The  following  section  presents  the  methods  we  employed  to  address  each  of  the 
identified  technical  objectives. 


Methods 

To  meet  our  first  objective,  of  developing  a  prototype  automated  tool  that  could  supply 
intelligent  feedback  for  performance  on  command  and  control  system,  we  performed  four  major 
tasks.  These  tasks  were: 

•  Develop  an  architecture  for  an  automated  tool 

•  Develop  a  prototype  version  of  the  tool 

•  Test  the  tool 

•  Determine  the  feasibility  of  incorporating  automatic  speech  recognition  into 
the  tool 

First,  we  developed  a  software  architecture  for  the  tool.  The  components  can  be  seen  in 
Figure  2.  The  components  consisted  of: 

•  a  data  filter 

•  an  expert  model 

•  a  comparator 

•  a  Bayesian  network  and  assessor 

•  an  output  report 

•  an  interface. 
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Figure  2.  Relationships  among  APAD  Components 


From  the  figure,  one  can  see  that  operators  would  generate  behavioral  data  in  response  to 
a  situation.  They  would  respond  to  the  situation  using  a  command  and  control  system.  The 
operator  performance  data  would  then  be  fed  to  a  comparator  that  would  compare  the  operator 
performance  to  the  performance  of  an  expert  model  responding  to  the  same  events.  The  results 
of  the  comparison  would  be  submitted  to  a  Bayesian  network  that  reflects  the  heuristics,  decision 
types,  and  cognitive  competencies  associated  with  the  tasks  that  need  to  be  performed  by  the 
operator  in  response  to  the  situation.  The  prior  probabilities  existing  in  the  Bayesian  network 
were  modified  based  on  the  performance  differences  between  the  model’s  performance  and  the 
actual  operator  data.  The  new  probabilities  were  to  be  fed  to  a  database  and  then  placed  into  a 
report  that  could  be  used  to  supply  feedback  to  the  operator(s). 

Development  of  each  component  is  discussed  in  the  following  paragraphs. 


Expert  Model 

To  develop  the  expert  mode,  we  first  selected  a  command  and  control  task  that  would  be 
appropriate  to  the  test  bed  that  we  were  using.  As  our  testbed  command  and  control  workstation, 
we  used  the  Systems  Training  Assessment  Research  Simulator  (C^STARS)  at  Brooks  AFB. 
This  facility  is  managed  by  the.  Air  Force  Research  Laboratory  Human  Effectiveness  Warfighter 
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Training  Research  Directorate  (AFRL/HEAB)  (now  AFRL/HRM),  from  whom  we  have  received 
permission  for  its  use  and  access  to  supporting  materials  and  research  data.  This  is  a  high- 
fidelity  simulator  used  for  fesearch  and  training  of  AW  ACS  weapons  directors  (WDs).  The 
simulator  is  configured  to  support  up  to  four  WDs.  There  are  also  workstations  for  personnel 
who  supply  the  WDs  with  aircraft  pilot  input  during  an  exercise.  The  C^STARS  has  a  few  minor 
discrepancies  from  actual  AW  ACS  WD  workstations.  However,  the  simulator  allows  for 
extensive  data  capture,  performance  analysis,  and  scenario  control.  These  features  ensured  that 
we  would  have  an  adequate  testbed  to  supply  us  with  appropriate  data  for  development  of  our 
strategy  for  intelligent  coaching  and  a  location  for  assessing  the  developed  coaching 
mechanisms. 

The  task  focus  for  our  effort  was  on  complex  information  gathering,  utilization,  and 
dissemination  in  a  team  environment.  To  briefly  describe  the  command  and  control  tasks 
performed  by  WDs:  WDs  work  in  teams  of  three  to  six  individuals,  depending  on  the  demand 
characteristics  of  the  operation.  They  are  responsible  for  managing  air  resources  during  air 
operations.  WDs  call  up  air  assets,  direct  them  with  regard  to  ways  to  accomplish  the  mission, 
schedule  them  for  re-fueling,  etc.  They  make  tactical  decisions  as  individuals  and  as  a  team 
(Elliott,  et  al.,  1997).  Therefore,  WDs  are  exemplary  of  command  and  control  teams. 

For  our  initial  feasibility  study,  we  selected  the  task  of  the  initial  committing  of  a  High 
Value  Air  Asset  (HVAA)  to  an  enemy  air  target.  To  determine  the  cognitive  steps  and  heuristics 
employed  by  an  expert  WD  toward  solying  this  problem,  we  interviewed  a  WD  domain  expert, 
Mr.  Mathieu  Dalrymple,  of  Veridian  Corp.  From  our  discussions  with  Mr.  Dalrymple,  we 
generated  an  expert  model  (seen  in  Appendix  A)  to  depict  the  heuristics  he  described  to  us. 

We  implemented  the  expert  model  in  MicroSaint,  a  commercial  off-the-shelf  tool 
developed  by  MA&D.  To  test  the  model,  we  re-coded  a  portion  of  an  existing  C^STARS 
training  and  research  scenario  so  that  it  could  be  included  as  events  in  MicroSaint  to  which  the 
expert  model  could  respond.  The  model  was  modified  until  it  responded  to  the  events  in  the  way 
a  real  expert  would. 


Operator  Data  Filter 

The  C^STARS  captures  operator  performance  data,  as  operators  respond  to  events 
presented  to  them  on  their  workstations.  The  events  are  inserted  into  the  workstation  via 
scenario  files.  The  events  appearing  in  the  scenarios  occur  on  a  timeline,  with  the  operators 
controlling  the  performance  of  their  own  assets  via  key  press  and  verbal  commands  to 
researchers  playing  the  roles  of  pilots,  and  who  are  using  other  workstations. 

The  simulator  captures  all  key  presses  performed  by  both  the  operators  and  the  simulated 
pilots.  However,  the  simulator  output  filter  routines  can  be  modified  to  selectively  extract 
desired  types  of  data.  Veridian  Corp.  developed  modifications  to  the  output  filter  routines  such 
that  only  commit  and  air  asset  vectoring  information  was  produced.  These  data  were  in  ASCII 
format.  An  additional  filter  was  developed  to  select  only  operator  data  that  contained  the  first 
commit  actions  for  fnendly  HVAAs.  Thus,  simulated  pilot  data  and  data  reflecting  HVAA 
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commits  subsequent  to  the  first  one  were  removed  from  the  data  to  be  compared  to  the  expert 
model’s  behavior.  Additionally,  a  routine  was  developed  to  reformat  the  data  so  that  it  can  be 
compared  to  the  expert  model  output.  A  routine  for  pasting  the  filtered  operator  data  into  a  table 
within  the  database  application  that  serves  to  connect  the  pieces  of  the  tool  was  developed. 


Comparator 

The  comparator  component  of  the  tool  resides  within  MicroSaint  as  an  additional 
function  that  is  performed  after  the  expert  model  completes  its  processing  of  the  events  to  which 
it  must  respond.  The  model  was  developed  so  that  its  output  is  saved  in  an  ASCII  file.  Then, 
using  the  programming  language  available  in  MicroSaint,  a  task  node  was  developed  that  uses 
the  operator  data,  as  an  event  stream  and  compares  it  to  the  expert  model  output  and  indicates 
which  expert  and  operator  outcomes  match  or  do  not  match.  The  compared  data  are  committed 
HVAAs,  their  targets,  and  the  vectors  to  be  taken  to  the  targets.  The  results  of  the  task  are 
written  to  an  ASCII-formatted  results  file. 


Bayesian  Network  and  Assessor 

Dr.  Kurt  VanLehn  and  his  associate,  Zhendong  Niu,  both  of  the  University  of  Pittsburgh, 
de\  eloped  the  Bayesian  network  and  assessor.  To  develop  the  Bayesian  network,  shown  in 
Figure  3,  the  tasks  in  the  expert  model  and  competencies  associated  with  those  tasks  were 
examined.  (The  competencies  were  developed  through  discussions  with  Mr.  Dalrymple).  The 
relationships  among  overt,  measurable  behaviors,  cognitive  tasks,  and  competencies  were 
graphed.  Each  node  in  the  network  could  be  in  either  of  two  states  (on/off  or  yes/no),  and  a 
prior  probability  was  assigned  to  each  node  for  each  state.  Initially  all  prior  probabilities  were 
set  at  .5.  However,  with  more  input  from  a  domain  expert  as  to  the  greater  or  less  likelihood  of 
any  state  in  a  node  being  greater  than  the  other  state,  the  prior  probabilities  can  be  so  adjusted. 

The  assessor  was  designed  to  accept  the  results  of  the  expert-operator  comparisons  on 
obscrx  able  measures,  and  then  modify  the  probabilities  within  the  network  based  on  the 
differences.  For  example,  if  the  operator  did  not  perform  an  observable  task  that  the  expert 
would  have,  the  assessor  changed  the  “task  observed”  node  to  zero  and  the  “task  not  observed” 
node  to  1 .00.  This  value  would  then  impact  all  of  the  nodes  related  to  it,  changing  their  values. 
0\  er  repeated  tasks,  the  probabilities  for  the  observable  tasks  are  modified  based  on  the  number 
of  same  or  different  performances  of  the  operator  versus  the  expert. 

Finally,  the  assessor  generates  an  ASCII  file  containing  the  node  name  and  the  final 
probability  associated  with  the  node.  MA&D  developed  Visual  Basic  code  to  paste  this  file  into 
an  MS-ACCESS  database  table.  For  further  details,  the  C++  code  for  the  assessor  may  be  found 
in  Appendix  B. 


9 


Figure  3.  APAD  Bayesian  Net 


Feedback  Report 

MA&D  designed  an  MS-ACCESS  report  to  display  the  results  of  the  assessor.  This  report 
presents  the  node  name  and  final  probability  associated  with  it,  of  all  the  competency  and 
hueristics-related  nodes.  This  report  was  designed  to  supply  operators,  instructors,  and  senior 
personnel  with  information  concerning  strong  and  weak  areas  of  competency.  This  information 
is  also  of  value  to  researchers  developing  process  measures  of  performance,  because  the  results 
allow  them  to  have  a  measure  of  implicit  behaviors.  An  example  of  this  report  is  shown  in 
Figure  4.  The  report  displays  the  Bayesian  network  node  number,  the  name  of  the  competency 
type,  the  probability  that  the  operator/team  shows  a  high  level  of  competency,  and  the  probability 
that  the  operator/team  show  a  low  level  of  competency. 


APAD  Diagnosis 


Net  Node  #  Competency  Type  Prob.  of  High  Prob.  of  Less 


1 

Monitoring 

0.5 

0.5 

2 

Categorization 

0.5 

0.5 

3 

ROE^Use 

0.5 

0.5 

4 

Sym  bology_knowledge 

0.5 

0.5 

5 

Data^gathering 

0.54704 

0.45296 

6 

7 

System_use 

Distance_calculation_from_data_or_ 

0.54704 

0.58258 

0.45296 

0.41742 

8 

Computational_skills 

0.58258 

0.41742 

9 

VisuaLcomparison 

0.58258 

0.41742 

10 

SituationaLawareness 

0.016044 

0.983956 

11 

Planning 

0.5 

0.5 

12 

Know!edge_of_heuristic 

0.012504 

0.987496 

13 

Aircraft_selection 

1 

0 

14 

Decision_making 

0.54704 

0.45296 

15 

Knowledge_of_assets 

0.960934 

0.039066 

16 

Knowledge_of_rules_for_HVAA_us 

0.999621 

0.000379 

Figure  4.  APAD  Bayesian  Results  Report 


Interface 


MA&D  developed  the  interface  for  the  tool,  named  the  AW  ACS  Performance 
Assessment  and  Diagnostic  (APAD)  system  in  Visual  Basic  (VB).  The  opening  screen  for  the 
tool  may  be  seen  in  Figure  5.  The  interface  was  developed  to  be  easy  to  use  and  to  require  a 
small  amount  of  data  entry.  Some  of  the  APAD  interface  screens  may  be  seen  in  Figures  6  and 
7.  The  VB  code  also  served  as  the  integration  product  for  the  other  components,  calling  the 
commercial-off-the-shelf  (COTS)  products  that  were  used  to  build  the  tool. 


Figure  5.  APAD  Opening  Screen 
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Figure  6.  APAD  New  Analysis  Operator  Data  File  Entry  Screen 


Run  Analysis 


Perth  to  Micro  Saint  |E  AProgram  Fiies\M^SMiao  Saint  3. 0\sakV.exe 

Path  to  MicroSaint  Modal:  |E  ;\Program  FilosSMAD \M icfo  S oirt  aOSmodds 


[El 


|Wd0201.mod 


Pun  Micro  Saint  Model 


j^ose 


) 

Figure  7.  APAD  Analysis  Screen 


Automatic  Speech  Recognition 

Due  to  the  highly  communicative  nature  of  many  WD  tasks,  we  undertook  to  examine  the 
feasibility  of  integrating  automatic  speech  recognition  (ASR)  as  a  way  to  capture  a  greater 
amount  of  performance  data.  The  examination  of  ASR  technology  as  it  could  be  applied  to 
automatic  performance  assessment  and  diagnosis  resulted  in  a  draft  working  paper,  which 
appears  in  Appendix  C. 
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Results 


APAD  Testing 

APAD  was  constructed  as  described  above.  Having  it  process  two  levels  of  operator  data 
tested  its  functional  capabilities.  The  first  set  was  constructed  to  by  modifying  the  output  of  the 
expert  model’s  performance  in  relation  to  the  test  scenario.  These  data  were  used  to  ensure  that 
the  comparison  process  and  the  assessor  were  functioning  properly  and  that  reasonable  and 
understandable  output  was  being  generated. 

The  second  test  data  set  was  derived  from  actual  operator  data.  These  data  were  from  an 
actual  C^STARS  run  on  a  longer  version  of  the  scenario  that  we  were  using.  The  data  contained 
operator  (both  fnendly  and  hostile)  HVAA  commits,  the  targeted  aircraft,  and  the  vectoring 
information.  The  data  file  was  reduced  in  length,  since  it  reflected  a  longer  version  of  the 
scenario  than  what  was  to  be  processed  by  the  model.  Additionally,  the  aircraft  designators 
appearing  in  the  file  were  double-checked  to  ensure  that  only  aircraft  that  were  in  the  reduced 
version  of  the  scenario  appeared  in  the  data  file. 

The  second  data  set  was  used  to  test  the  complete  functionality  of  APAD.  The  interface 
required  the  user  to  indicate  the  path  to  the  data  file.  The  data  file  was  accessed  and  parsed  by 
the  tool.  Then  the  user  indicated  the  location  of  the  model  and  of  MicroSaint.  The  tool  opened 
MicroSaint  and  placed  the  model  and  the  operator  data  into  it.  The  user  then  ran  the  model  to 
create  the  results  and  comparison  files  to  be  used  by  the  assessor.  Once  the  model  had  finished 
running,  the  user  needed  to  close  the  application,  before  continuing.  Then  the  user  selected  “Run 
Bayesian  Analysis.”  At  this  point,  the  tool  accessed  the  network  and  the  assessor  code.  The 
assessor  entered  the  information  in  the  comparison  file  into  the  Bayesian  network  and  recorded 
the  final  probabilities.  The  VB  interface  pasted  the  assessor  results  file  into  an  MS-ACCESS 
database  table.  The  user  was  able  to  call  up  the  report  of  the  Bayesian  results.  The  results  were 
then  examined  for  their  plausibility,  given  the  contents  of  the  operator  data  file. 

The  Bayesian  output  for  the  second  data  set  indicated  that  there  was  a  weakness  in  the 
model  with  regard  to  its  reflection  of  the  type  of  data  presented  to  it.  It  should  be  noted  that  the 
model  only  addressed  initial  commit  actions.  However,  the  data  included  both  initial  and 
subsequent  commit  actions  for  the  same  HVAAs.  Since  the  expert  model  did  not  take  into 
account  aircraft  location  and  availability  after  initial  commits,  anytime  an  operator  committed  a 
previously-committed  HVAA,  because  of  its  location,  to  a  hostile  air  contact,  there  would  be  a 
flagged  discrepancy  between  the  model  and  the  operator  data.  Thus  the  analysis  was  not  totally 
accurate  because  of  the  actual  spatial  relationships  among  aircraft  when  the  operators  committed 
them.  The  resulting  inaccuracy  does  not  point  to  a  design  or  feasibility  problem,  rather  just  a 
scope  of  task  coverage  issue. 
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Conclusions 

Feasibility  of  Use  of  Bayesian  Technology  for  the  Development  of  Real  Time 
Intelligent  Coaching 

After  evaluating  the  output  of  APAD  and  demonstrating  its  capabilities  to  personnel  at 
ARL/HRM,  we  concluded  that  a  Bayesian  approach  to  automatic  performance  assessment  and 
diagnosis  is  feasible,  if  we  ensure  fairly  complete  coverage  of  tasks.  Greater  representation  of  the 
tasks  to  be  performed,  including  subsequent  commits,  would  alleviate  problems  that  are 
associated  with  restricted  task  coverage. 

Additionally,  we  determined  that  our  architecture  would  not  be  transferable  to  operational 
systems  in  which  the  environmental  events  are  not  known  prior  to  actual  task  performance. 

Thus,  in  Phase  II  of  this  effort,  we  intend  to  develop  a  filter,  in  Visual  Basic,  that  will  take  a 
workstation’s  log  or  environment  file  (which  it  captures  during  operation),  select  the 
environmental  events  of  importance,  and  paste  them  into  the  events  queue  of  the  MicroSaint 
model.  In  that  way,  the  model  will  be  able  to  run  against  events  from  an  operational  system— 
the  same  events  responded  to  by  the  operator  of  the  workstation.  This  would  also  alleviate  the 
need  for  the  expert  model  to  calculate  the  positions  of  the  aircraft — the  environment  file  would 
supply  these  data. 

One  aspect  of  tool  development  that  we  have  determined  that  requires  modification  is  the 
procedure  by  which  the  Bayesian  network  is  generated.  For  the  effort  presented  here,  it  was  ^ 
produced  by  hand  after  examination  of  the  expert  model  and  the  paths  down  which  the  expert  s 
solution  could  traverse,  given  different  environmental  events.  In  subsequent  work,  we  plan  to 
develop  a  Bayesian  network  generator  which  will  take  as  its  input  an  expert  model  and  all  of  its 
potential  solutions  and  will  output  a  developed  network. 

Finally,  we  realize  the  need  for  greater  testing  of  the  tool’s  usability  and  extension  of  the 
feedback  methods.  We  can  predict  that  such  testing  will  indicate  the  need  to  develop  stronger, 
more  concrete  and  directive  feedback  and  guidance  in  order  to  ensure  that  operators,  supervisors, 
and  training  personnel  will  be  able  to  make  functional  use  of  the  information  supplied  by  the 
tool.  We  intend  to  gather  input  from  the  potential  user  community  to  determine  the  format  and 
level  of  guidance  most  appropriate  to  the  proposed  uses  of  APAD. 

Our  final  assessment  is  that  the  approach  that  we  selected  for  generating  intelligent 
information  to  be  used  for  intelligent  coaching  is  promising.  However,  the  information 
generated  by  the  Bayesian  analysis  needs  to  tied  to  concrete  remediation  and  guidance  that  can 
be  used  by  operators,  supervisors,  or  training  personnel.  Additionally,  we  did  not  strive  to 
supply  “real  time  coaching,”  in  the  sense  that  the  coaching  occurred  during  job/task 
performance.  During  task  performance,  coaching  may  not  be  what  operators  need;  remedial 
guidance,  as  implied  the  term  “coaching”  may  detract  from  operational  performance.  Thus,  we 
argued  that  such  guidance  should  be  supplied  subsequent  to  the  operator’s  completion  of  the  task 
set  or  mission.  However,  during  a  mission  or  task  set  performance,  operators  may  benefit  from 
intelligent  aids  that  support  their  decision  processes,  such  as  highlighting  critical  information. 
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ASR  Feasibility 

After  examining  the  current  state  of  ASR  technology,  we  cautiously  concluded  that  this 
technology  may  have  utility  within  a  performance  assessment  tool.  We  feel  that  much  of  the 
ASR  technology  is  close  to  the  state  where  it  can  be  integrated  into  team  performance  and 
communications  assessment  tools.  The  important  word  here  is  integrated.  There  is  ASR 
technology  available  that  will  meet  many  of  the  requirements  for  use  in  a  tool  such  as  AP AD. 
However,  work  must  be  undertaken  to  bring  these  pieces  together. 

For  example,  the  state  of  the  technology  is  such  that  multiple  speaker  recognition  is 
currently  available  in  conference  transcription  software,  and  on-going  research  shows  great 
promise  for  this  technology.  Additionally,  work  in  environmental  noise  removal,  or  reduction, 
seems  to  be  moving  swiftly  such  that  the  capability  should  be  available  in  an  ASR  tool  in  the 
very  near  future.  ASR  technology  already  supports  continuous  speech  recognition  and  relatively 
easy  methods  for  training  voice  files,  with  profiles  being  updated  after  every  session  with  the 
user.  Finally,  there  are  programmer  tool  kits  to  allow  ASR  technology  to  be  integrated  into  other 
applications.  (Possibly,  such  an  application  kit  would  allow  for  the  integration  of  ASR 
technology  with  a  time-stamping  capability  needed  to  synchronize  other  captured  performance 
data  with  the  verbal  acts.)  These  pieces  need  to  be  brought  together  in  a  single  package  before 
one  can  begin  to  integrate  it  with  other  components  for  comprehensive  automated  performance 
assessment,  job  aiding,  or  training  tools. 

Thus,  our  overall  recommendation  is  to  introduce  such  a  capability  into  APAD,  or  a 
similar  tool,  to  better  see  how  much  additional  information  it  supplies  to  the  diagnostic  process. 
If  it  adds  substantially  to  the  process,  as  we  think  it  will,  then  the  capability  for  voice  data 
capture  and  analysis  should  be  fully  integrated  into  the  tool. 


Future  Work 

In  Phase  I  of  this  effort,  we  demonstrated  the  feasibility  of  using  Bayesian  networks  to 
generate  feedback  for  post-event  reflection.  To  extend  this  work  and  to  meet  an  overall  objective 
of  a  stand-alone  automatic  debriefing  tool/coach,  which  WD  personnel  can  use  subsequent  to  a 
mission  phase  or  training  exercise,  we  would  need  to  meet  six  objectives. 

1.  Increase  task  coverage  of  the  APAD  tool. 

2.  Use  environment  files  to  supply  event  data. 

3.  Integrate  automatic  speech  recognition  (ASR)  technology  to  augment  operator  data  files. 

4.  Incorporate  visual  playback  capabilities  and  summary  performance  statistics. 

5.  Extend  and  expand  the  capabilities  of  the  Bayesian  network. 
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6.  Determine  the  measures  of  performance  for  such  tools  in  order  to  determine  their  actual 

effectiveness. 

Each  of  these  objectives  are  discussed  below. 

Increase  task  coverage  of  the  APAD  tool,  jjjg  current  version  of  APAD  is  a 
prototype,  designed  to  prove  the  concept  that  Bayesian  networks  can  be  used  to  generate 
intelligent  feedback  for  systems.  Since  the  tool  was  a  feasibility  demonstration,  it 
incorporated  only  one  domain  task.  Now  that  the  feasibility  of  the  concept  has  been  shown,  the 
tool  can  be  extended  to  cover  a  wide  range  of  mission  critical  WD  tasks.  Extending  the  task  base 
will  allow  us  to  move  the  tool  from  a  concept  stage  to  a  product  for  potential  fielding.  To  that 
end,  we  will  expand  the  expert  model  and  the  Bayesian  network  to  cover  all  mission-critical  WD 
tasks,  such  as  aircraft  re-fueling  and  tanker  control  tasks,  bomber  control,  and  initial  track 

control.  We  will  also  incorporate  tasks  that  are  based  on,  or  augmented  by,  verbal 

communications.  Since  many  tasks  performed  by  the  WDs  require  verbal  communications,  any 
model  of  their  performance  that  does  not  include  verbal  tasks  cannot  accurately  portray  WD 
mission-critical  behavior.  Inclusion  of  verbal  tasks  will  allow  us  to  address  all  of  the  important 
WD  tasks,  not  a  small  subset.  Additionally,  since  environments  frequently  include  verbal 
performance  by  operators,  if  an  APAD-like  tool  is  to  be  fielded  into  other  situations,  the  ability 
to  include  verbal  tasks  in  the  model  is  of  paramount  importance. 

Additionally,  we  will  develop  our  model  so  that  it  supports  the  expanded  capabilities  of 
the  Bayesian  network  that  we  plan  to  undertake.  To  do  so,  we  will  build  the  model  so  that  does 
not  take  actions,  per  se,  but  only  recommend  them.  Moreover,  it  will  be  non-deterministic  and 
buggy,  in  that  if  multiple  derivable  actions  can  be  derived  via  correct  or  incorrect  reasoning,  it 
will  recommend  them  all.  The  model  will  record  the  reasoning  associated  with  recommended 
actions,  and  these  will  be  converted  into  a  Bayesian  network. 

Use  environment  files  to  supply  event  data,  the  current  version  of  APAD,  the 
environment  to  which  operators  responded  was  specified  by  training  scenario  files  that  were  used 
to  stimulate  the  expert  model  and  to  which  actual  operators  could  respond  within  the  C  STARS 
milieu.  Since  the  C^STARS  is  a  training  and  research  device  rather  than  an  operational 
AWACS,  its  operation  is  stimulated  by  scenario  event  files  and  simulated  enemy  pilot  responses 
to  system  users.  Operational  AWACS,  on  the  other  hand,  functions  in  response  to  an  actual 
detected  environment.  In  order  to  be  useful  within  an  operational  environment,  APAD  must  be 
able  to  gather  environment  data  from  a  functioning  AWACS  during  a  mission  or  portion  of  a 
mission.  Then  APAD  must  use  these  data  to  stimulate  the  expert  model,  so  its  responses  may  be 
compared  to  that  of  the  operational  team.  Figure  8  displays  the  data  flow  from  the  C  STARS  to 
a  revised  APAD  (R-APAD). 
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Figure  8.  Data  Flow  from  C^STARS  to  R-APAD 


To  meet  this  objective,  we  will  use  environment  snapshot  files  retrieved  from  the 
C'^STARS.  These  same  types  of  files  are  generated  and  saved  by  an  AWACS  as  history  or  log 
files  during  its  operation.  These  files  contain  a  snapshot  of  the  system  state;  these  snapshots  are 
collected  every  three  seconds  during  system  operation.  To  use  these  files,  we  will  build  a  filter  to 
selectively  extract  the  information  that  we  need  to  supply  the  model.  We  will  also  build  an 
interface  to  the  model  in  order  to  supply  it  with  the  necessary  information.  The  model  will  then 
be  able  to  respond  as  it  would  in  an  operational  setting. 

Integrate  automatic  speech  recognition  (ASR)  technology  to  augment  operator  data 
files.  •]  o  be  able  to  compare  the  performance  of  the  model  on  speech  tasks  with  the  operators’ 
performance,  we  need  to  be  able  to  collect  and  analyze  operator  speech  data.  To  accomplish 
this,  we  will  program  an  ASR  interface  that  will  capture  WD  utterances,  time-stamp  them,  and 
place  them  into  a  file  that  can  be  interleaved  with  the  operator  data  file  currently  generated  by 
C'STARS. 

We  will  also  include,  if  possible,  the  capability  to  use  APAD  to  supply  correct,  digitized 
versions  of  speech  data,  as  part  of  APAD’s  use  as  a  de-brief  tool.  The  aim  is  to  supply  the 
operator  with  a  correct  speech  model  for  later  use. 
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Incorporate  visual  playback  capabilities  and  summary  performance  statistics.  One 

important  method  of  supplying  feedback  that  we  have  found  through  our  research  on  command 
and  control  tasks  is  that  of  visual  playback  of  the  important  events  to  which  verbal  feedback  is 
referring.  In  that  way,  the  operator  can  mentally  place  him  or  herself  back  into  the  situation  to 
perform  a  self-assessment  in  conjunction  with  the  automated  feedback.  It  is  similar  to  re-living 
the  events  to  pinpoint  the  exact  situation  in  which  alternative  decisions  may  have  been  more 
appropriate. 

To  meet  this  objective,  we  will  program  into  the  user  interface  the  ability  to  present  on  a 
scope-like  display  the  positions  of  critical  tracks,  and  other  information,  in  relation  to  time  within 
the  mission.  We  will  also  make  available  summary  statistics,  such  as  number  of  successful 
commits,  kills,  etc.  to  give  the  operators  a  full  picture  of  their  performance.  Frequently,  these  are 
the  types  of  data  that  operators  find  to  be  important  in  their  assessment  of  themselves.  These 
outcome  data  are  items  that  are  typically  reported  during  assessment. 

Additionally,  we  will  tie  the  output  from  the  Bayesian  assessment  more  closely  to 
guidance  for  performance  change.  During  Phase  I,  we  supplied  a  report  concerning  competency 
levels,  based  on  the  outcome  of  the  Bayesian  analysis.  In  subsequent  work,  we  intend  to  tie 
those  outcomes  to  constructive  guidance. 

Finally,  we  need  to  develop  a  user  interface  that  will  allow  operators  the  ability  to  receive 
feedback  subsequent  to  mission  or  training  performance.  This  interface  should  be  easy-to-use, 
and  the  product  will  require  very  little  training  to  use. 

Extend  and  expand  the  capabilities  of  the  Bayesian  network,  Bayesian  network 
described  in  this  report  was  developed  specifically  for  our  target  task.  We  will  need  to  expand 
the  network  to  include  new  tasks.  During  Phase  1,  the  network  was  built  by  hand.  As  mentioned 
earlier,  we  intend  to  develop  a  method  to  automatically  generate  a  complex  and  extensive 
Bayesian  network,  using  the  expert  model.  To  address  this  issue,  we  will  build  an  expert  model 
that  will  not  take  actions,  but  only  recommend  them.  It  records  its  reasoning,  which  we  convert 
to  a  Bayesian  net.  The  links  lead  from  input  events  in  the  history  file  (e.g.,  observing  a 
potentially  hostile  air  contact  cross  into  friendly  air  space)  to  state  variables  (e.g.,  the  air  contact 
is  presumed  hostile)  to  actions  (e.g.,  directing  a  friendly  squad  to  intercept  it). 

For  each  real  action  in  the  histoty  file,  there  may  be  hundreds  of  actions  predicted  by  the 
model.  The  larger  variety  of  potential  actions  occurs  because  there  is  genuine  ambiguity  in  the 
task  (e.g.,  if  two  squads  are  tied  for  closest  to  the  hostile,  either  can  be  committed  to  an 
intercept),  there  is  uncertainty  (e.g.,  it’s  not  clear  how  much  fuel  a  friendly  has),  and  there  are 
errors  (e.g.,  the  operator  is  confused  about  the  weapons  on  board  the  fnendly  chosen  for 
committing).  Our  goal  will  be  to  have  so  many  actions  predicted  by  the  model  that  there  is 
rarely  an  action  in  the  history  file  that  is  not  predicted. 

When  processing  the  operator  and  environment  files,  predicted  actions  will  be  identified 
as  Early,  On-time,  Late  or  Never.  Most  actions  will  be  identified  "Never."  The  nodes 
corresponding  to  input  events  are  not  scheduled  for  update  action,  but  instead  are  given  a  prior 
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probability  corresponding  to  how  likely  the  operator  is  to  have  observed  that  information  given 
the  current  settings  on  the  console  and  the  task  load. 

What  will  make  this  approach  feasible  for  automatic  construction  is  that  we  will  assume 
that  all  tasks  are  instances  of  generic  tasks,  such  as  intercepts  and  walking  the  clock.  Thus,  the 
networks  can  either  be  built  in  advance  in  generic  form  and  adapted  to  the  specific  situation  in 
the  history  file  (e.g.,  connecting  it  to  a  node  representing  the  AC's  time  in  the  air,  a  node 
representing  the  currents  ROE,  etc.). 

The  resulting  network  will  span  the  whole  history  file.  Potentially,  there  will  be 
thousands  of  input  events  and  tens  of  thousands  of  predicted  actions  (of  which  hundreds  are 
observed  actions).  Since  this  will  result  in  a  very  large  network,  it  will  be  updated  in  piece-meal 
fashion,  but  the  overall  outcome  will  be  a  network  that  is  very  sensitive  to  real  and  expected 
operator  actions. 

Determine  the  measures  of  performance  for  such  tools  in  order  to  determine  their 
actual  effectiveness.  Although  research  and  commonsense  suggest  that  supplying  intelligent 
feedback  to  personnel  will  improve  their  subsequent  performance,  we  need  to  establish  this  as  a 
fact.  To  do  this,  we  will  develop  methods  to  assess  change  in  performance  subsequent  to  receipt 
of  feedback. 

Initially,  we  will  use  methods  that  have  been  used  in  other  training  tool  assessments.  In 
general,  we  will  compare  the  performance  of  WD  teams  who  do  not  receive  feedback  to  those 
that  do  receive  feedback  both  prior  to,  and  subsequent  to,  the  feedback.  All  teams  will  function 
within  the  same  operational  or  scenario  environment.  However,  if  this  approach  does  not  meet 
our  requirements,  due  to  the  nature  of  the  environment,  we  will  investigate  other  assessment 
methods,  such  as  surveys  addressing  the  perceived  utility  of  the  feedback  or  analyses  of  process 
change.  (It  is  very  possible  that  for  journeymen  level  personnel,  the  fine-tuning  of  performance 
that  results  from  intelligent  feedback  may  be  too  subtle  to  detect  using  standard  product 
measures  of  performance  change.  In  this  case,  we  will  need  to  determine  appropriate  measures 
of  process  change.) 

Once  a  stand-alone  tool  has  been  developed,  we  can  begin  to  further  extend  the  product. 
There  are  several  paths  to  product  commercialization  that  may  be  taken.  First,  APAD  could  be 
included  as  an  integral  component  of  the  AWACS.  Second,  the  APAD  architecture  could  be 
expanded  to  encompass  other  domains  such  as  the  AEGIS  combat  information  center  (CIC), 
air  traffic  control  and  military  or  rescue  forces  coordination. 

Finally,  the  APAD  architecture  and  process  could  become  the  basis  for  the  development 
of  an  authoring  tool  for  easy  development  of  future  automated  evaluation  and  diagnostic 
products.  We  foresee  the  need  for  different  tools  to  support  development  of  intelligent  coaches 
for  command  and  control. 

One  authoring  tool  would  be  specific  to  training  situations,  in  which  scenario  ground 
truth  can  be  determined  prior  to  use.  To  give  this  tool  its  greatest  utility,  it  should  be  designed  to 
integrate  the  development  of  the  assessment  tool  and  training  scenarios.  This  means  developing 
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interfaces  to  scenario  development  modules  for  training  workstations.  One  can  conceive  of  an 
interface  that  would  allow  the  individuals  building  scenarios  to  build  as  much  of  the  expert 
models  as  possible  conciurent  with  exercise  development.  As  the  developer  identifies  expected 
(of  expert)  responses  and  options  to  scenario  events,  information  sources,  etc.,  these  same  data 
would  be  supplied  to  a  module  that  would  build  the  intelligent  coach.  This  interface  for 
constructing  models  would  be  supported  by  a  final  authoring  module  that  would  support  easily 
constructing  the  Bayesian  networks  necessary  for  the  assessment  module. 

The  second  authoring  tool  would  share  components  with  the  first.  However,  it  would 
differ  in  that  it  would  be  used  to  set  up  general  structures  for  capturing  environmental  events,  as 
they  happen,  not  before  the  fact,  as  in  the  tool  for  training  development.  As  with  the  authoring 
tool  for  training  workstations,  however,  this  tool  would  allow  a  domain  expert  to  input  the 
structures  for  the  expert  models  and  the  information  needed  to  automatically  build  the 
components  of  the  assessment  module. 

The  steps  presented  above  would  result  in  a  final  product  or  set  of  products  that  would 
offer  managers  and  trainers  within  various  command  and  control  environments  a  way  to  develop 
a  customized  intelligent  coach  and  integrate  it  with  their  work  or  training  stations.  The 
customized  coach  would  then  collect  and  analyze  performance  data,  and  give  appropriate 
feedback  to  the  users  during  a  debriefing  session.  The  users  would  be  able  to  use  this  well-tuned 
feedback  to  modify  their  subsequent  behavior,  thereby  resulting  in  better  performance. 
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Appendix  A:  Expert  Model 


The  following  pages  contain: 

•  A  visual  representation  of  the  top  level  APAD  expert  model 

•  A  visual  representation  of  the  “Transitional  Mission”  level  of  the  APAD  expert 
model,  because  the  scenario  that  was  used  in  APAD  was  a  transitional  one;  these 
graphics  represent  a  left  to  right  progression 

•  The  complete  code  file  for  the  APAD  expert  model 
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APAD  MicroSaint  Expert  Model:  Graphic  Representations 


Figure  A-1.  Top  Level  APAD  Expert  Model 


Figure  A-2.  APAD  Expert  Model,  Part  1 
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Figure  A-3.  APAD  Expert  Model,  Part  2 


Figure  A-4.  APAD  Expert  Model,  Part  3 
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APAD  Expert  Model  MicroSaint  Code 
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{ id  n  1 30  to  140  assume  friend } 

Competency:  Monitoring 


212 

213 

3 

5 

1 

1 

I 

1 

.  1 .  {6  seconds  1  - , 

4 

acjd(iag)==21 1  (  acjd(tag]===4l5;  {friendly  general  notspecified  -  we  are  using  for  HVAA  or  415  for  AWACS} 
ac  id[iag}==^213;  {friendly  general  nonmil  Civilian  -  we  are  using  for  COM  AIR} 


comp_  1=1, 
comp_2  =0, 
comp_3  =0, 


1 


1 

212 

Dcicnninc  HVAA 
3 
1; 


A-12 


Competency:  Categorization,  Symbology  knowledge 

0 

4 

5 
1 
1 
1 


1 

CLASS_MATCH; 

1 


if  Miss_typel[tag)==l  | 
Miss_type2[tag]=l 
then  tol_HVAA+=l, 
avail_HVAA+=l., 
HVAA[toLHVAA]  :=  tag; 

comp_l  :=2; 
comp_2:=4; 
comp_3  =0; 


1 

friend!)  non-HV’AA 

3 

1. 

Compcicncy  Categorization,  Symbology  knowledge 

0 

4 
(> 

I 

1 

1 


(  I  ASS  MA'K'H. 


comp  1  2. 

comp  2  -4. 
comp  ' 

1 

1 

220 

Mnlr  assm  host  susp 
3 

WI)  >0, 

Competency  Monitoring,  ROH  use 


221 

2 

3 

1 

1 

1 


1 

,2;  {12  seconds} 


4 

truncate(ac  Jd  [tag]/l  0)=1 4 ; 

{id  #  from  141  to  148  assume  host/susp} 

ac_id[tag]=311  &  hostile_act[tag]  =1;  {hostile} 


HOSTILE_ACT; 
if  hosliie_act[tag]  ==1 
then  acjd[tag]:=31 1; 

comp_l  -] ; 
comp_2:=3; 
comp_3:=0; 


1 

1 

221 

knovsTi  hostile 
3 

\\'D>0; 

Competency:  Calcgonzation,  Symbology  knowledge 
1 

222 

3 

3 

1 

1 

1 


1 

Cl-ASS_MATCH; 


1. 


\VI>-1, 


host_count^'=l , 

com_hoslile  Squad_ld(tag]; 

comp_l  :=2; 

comp_2:=4, 

comp_3:=0; 

TRACI:; 

1 

++-M-4- 

1 

222 

Det  pos  aircraft 


A-14 


3 

1; 

Competency:  Data  gathering,  system  use 


1 

223 

4 

1 

1 

1 

1 


I 

.5/60;  {half  a  second} 


2 

1; 


comp_l  :=5; 
comp_2:=6; 
comp_3:=0; 

TRACH;- 

1 

I 

223 

Compare  to  HVA As 
3 

avail_nVAA>0; 

Competency;  Distance  calculation  from  data  or  screen  view.  Computation  skills,  visual  comparison 


1 

224 

5 

i 

1 

1 

1 


i 

5  ()<!.  I  halt  a  second! 


comp  ! 
comp  2 
comp  3  -9. 

TRACI  . 

1 

1 

224 

Del  nrcsi  H\’AA 
3 
1; 

Competency:  Situational  awareness.  Planning 


A-15 


1 

225 

6 

I 

1 

1 

1 


1 

.5/60;  {half  a  second} 


2 

1; 

DET_N  BAREST; 

if  do_compare  then  COMP_HVAA; 


comp_I  :=10; 
comp_2:“ll; 
comp_3:=0; 

TRACE; 

1 

1 

225 

Determine  flVAA  fuel  level 

1. 

Compeicnc> :  Knowledge  of  heuristic.  Situational  awareness 


251 

226 

S 

I 

1 

1 

1 


1 

5  60.  {half  a  second} 


tiicl  ok'~0. 


hie  I 


[1*1  I  (Til  C  Kl. 

il  tuel_ok^=^0  then  occup|nearcst):=2,  avail_HVAA-=l; 

cimip  1  =12. 
comp_2  -10, 
comp  3  -0. 

TRACE. 


1 

I 

226 


A-16 


Fuel  level  okay 
3 
1; 

Competency:  Knowledge  of  heuristic,  Situational  awareness 
1 

227 

9 

1 

1 

1 

1 


1 


2 

1; 


comp_l  :=12; 
comp_2:=10: 
comp_3:=0; 
TRACE; 


1 

1 

227 

Dispatch  11\’AA 
3 
1; 

Competency:  Aircraft  selection 
Measure:  Commit  action 


1 

228 

10 

1 

1 

1 

1 


1 

spi:ak. 


1. 


com_h\  aa  1  =Squad_ld[ncarest); 
com_hvaa2  :=Squad_ld|ncarcst]; 


DISPATCH. 


compel  :=1 3 
comp_2:=0; 
comp_3:=0; 
TRACE; 


A-17 


1 

M  M  + 

1 

228 

Commit  to  target 
3 
1; 


Competency:  Decision-making 
Observable:  Commit  action  on  track 
1 

229 

11 

1 

1 

1 

1 


1 

PUSH_BUTTON; 


2 

1 


com_timc  :=  clock; 
commit_couni+=2; 

comp_l  :=14; 
comp_2;=0; 
comp_3:=0; 
TRACE, 


1 

229 

Re-estimaic  fuel  Ivl 

3 

I; 


240 

230 

12 

1 

1 

1 

1 


I 

.5  60.  I  half  a  second} 


4 

fucl_ok=-0; 
fucl_ok==  1 ; 


FUEL_CHECK2; 


A-18 


1 

+-H-H- 

1 

230 

OK  fuel;  asses  arms 
3 


Competency:  Knowledge  of  heuristic,  Situational  awareness 
2 

235 

231 

15 

1 

1 

1 

1 


1 

.5/60;  lhalf  a  second} 


3 

Miss_type2(ncarest]>0; 
Miss_i>pcl  (nearest  j>0; 


compl  =12, 
comp_2  =10. 
comp  3  =0. 
TRACI^ 


1 

1 

231 

radar  guided 

3 

!. 

fompctencN  Kno\s  ledge  of  heuristic,  Situational  aw^areness,  knowledge  of  assets 


! 

1 

1 

1 

1)1(11)1  . 

3 

1, 

1. 


A-19 


compel  :=I  2; 
comp_2:=10; 
comp_3:=15; 
TRACE; 


1 

1 

232 

Pursuit 

3 

1; 

Competency:  Knowledge  of  heuristic.  Situational  awareness 
1 

260 

17 

1 

1 

1 

1 


1 

DECIDE; 


1; 


vector^l; 
comp_l  :=12: 
comp_2:-10, 
comp_3:=0; 
TRACE; 


1 

1 

233 

Nose  approach 
3 
1. 

C'ompciency.  Knowledge  of  heunstic.  Situational  awareness 
1 

260 

17 

2 

I 

1 


1 

DECIDE; 


1; 


vector:=2; 


comp_l;=12; 

comp_2:=10; 

comp_3:=0; 

TRACE; 


] 

t  }  f ++ 

1 

235 

heat  seeking 
3 


Competency:  Knowledge  of  heuristic,  Situational  awareness,  knowledge  of  assets 
2 

237 

236 

16 

3 

1 

1 

1 


1 

DECIDIi; 

3 

1; 

1; 


compl  =12. 
comp_2:  =  iO, 
comp_3  =15, 
TRAC!.. 


1 

1 

236 

Pursuu 


1. 


CompcicncN  Knowledge  of  heuristic.  Situational  aw'areness 


1 

260 
I  ' 

3 

1 

! 

1 


1 

Dr.ciDi-:. 


2 

1; 


A-21 


vector:=l ; 

comp_l;=12; 

comp_2:=10; 

comp_3:=0; 

TRACE; 


1 

-H-H-+ 

1 

237 

Side  approach 

3 

1; 

Competency:  Knowledge  of  heuristic,  Situational  awareness 
1 

260 

17 

4 

1  . 

1 

1 


1 

OI-CIDE; 


vector  =3. 
comp_  1=12. 
comp_2  =10. 
comp_3  =0, 
TRACT', 


1 

1 

24t) 

i  ou  tuel,  asses  arms 

1. 


(  ompetenev  Knowledge  of  heuristic.  Situational  awareness 


.  2A] 
24 


1 

1 

1 


1 

.5  60,  I  half  a  second! 


3 

M 1  ss_t>pc2  ( nearest  ]>0; 


Miss_typel  [nearest]>0; 


comp_l:=12; 

comp_2:=10; 

comp_3:=0; 

TRACE; 


1 

++-H-+ 

1 

241 

heat  seeking  arms 

3 

I; 

Competency:  Knowledge  of  heuristic,  Situational  awareness,  knowledge  of  assets 
1 

242 
16 
6 

1 

1 

1 


1 

DECIDE, 


2 

1, 


comp) 

comp_2 

comp_3 

TRACIE 


=  12; 
=  10; 
=  15; 


1 

1 

242 

Stdc  approach 

3 

I. 

('ompcicncN  Knowledge  of  heuristic.  Situational  awareness 
1 

20(} 

17 

1 

1 

1 

1 

DECIDE; 


2 

1; 


A-23 


vector:=3; 

comp_l:=12; 

comp_2:=10; 

comp_3:=0; 

TRACE; 


1 

+-HH-+ 

1 

243 

radar  guided  arms 
3 


Competency:  Knowledge  of  heuristic,  Situational  awareness,  knowledge  of  assets 
1 

244 

16 

7 

1 

1 

1 


1 

DECIDE: 


2 

1; 


comp_l  :=12; 
comp_2:=l0: 
comp_3:=l  5: 
TRACE, 


1 

+-♦-+-*--1- 

1 

244 

Nose  approach 
3 
1; 

Competency  Know  ledge  of  heuristic.  Situational  aw-areness 
1 

260 

17 

7 

1 

1 

1 


1 

DECIDE; 


2 

1 


A-24 


vector:=2; 

comp_l”12; 

comp_2:=10; 

comp_3:=0; 

TRACE; 


1 

M  M  + 

I 

251 

Fuel  Ivl  too  low 
3 
1; 


Competency:  Knowledge  of  heuristic.  Situational  awareness 


1 

252 

9 

3 

1 

1 

1 


comp  1  “12.  . 
comp  2  =*1(). 
comp_3 
TRAC'F. 

1 

I 

252 

Find  nc\l  closest  MVAA 
avail  H\  AAXi. 

(  ompetenev  Siluaiional  awareness,  decision-making 
1 

25' 

s  ■ 

4 

I 

1 

1 

5  60.  I  hair  a  second ! 


1; 

DFT  NIiARFST, 


A-25 


/ 


comp_l  :=10; 
comp_2:=14; 
comp_3:=0; 
TRACE; 


1 

■H-+++ 

1 

253 

Det  posit  HVAA 

3 

1; 

Competency:  Data  gathering,  system  use 
1 

225 

9 

4 
1 
1 
1 


1 

.5/60;  {Haifa  second} 


2 

1; 


comp_l  :=5; 
comp_2:=6; 
comp_3:=0; 
TRACE; 


I 

260 

Vector  H\'.‘\A 
3 
1: 

Competency;  Knowledge  ofrulcs  for  HVAA  use  given  fuel  and  armament  status 

Obser\'eable  that  indicates  success  at  next  two  levels  up:  Measured  by  vector  taken  by  committed  HVAA  to  target 


600 

IS 

3 


1 

SPEAK; 


1; 

if  do_compare  then  index:=commit_count-l ,  CIE_DATA,  start(601 , 0); 


A-26 


WI>=1; 

comp_l:=16; 

comp_2:=0; 

comp_3:=0; 

TRACE; 

if  do_compare  then  index:=commit_count,  CIE_DATA,  start(601,0); 


^  I  I  I  I 
1 

600 

reset 

3 

1; 


0 

19 

3 

1 

I 

1 


1 


1 


com_hvaal  :=0; 
com_hvaa2:=0; 
com_hosii!e:=0; 
not_ncarest:=0; 
not_fuel  1  :=0, 

I 

1 

601 

Clf:  Data  Gathering 
3 
1; 


0 

4 

I 

I 

I 


1 


I 


4 

Wartime  mission 
0 
1, 


A-27 


0 

2 

2 

1 

1 

1 


115 

1 

-H-+++ 

1 

53 

Determine  position 
4 
1; 

Competency;  Data  gathering,  system  use 
1 

56 

3 

2 

1 

1 

1 


1 


1 

1 

54 

Determine  known  hostile 

4 

5, 

('ompetcncy:  C  alcgorizalion.  Symbology  knowledge 
1 


i 

i 

1 

I 


K 


1 

1 

55 

Determine  potent  hostile  air 


A-28 


4 

1; 

Competency:  Categorization,  Symbology  knowledge 
1 

57 

2 

5 
1 
1 
I 


1 

2 

1; 


I 

1 

56 

Determine  HVAA  armament 
4 
1; 

Competency;  Knowledge  of  assets 


101 

102 

4 


1 

1 

1 


1 

4 

1 .  ;armamcnt_t>T5e==l ; } 
1 ;  iarmameni_iypc==2;  1 


1 


Dciemime  position 
4 
1. 


CompeicncN  ■  Data  gathering,  system  use 
1 

58 

3 

4 
1 
1 


A-29 


1 


1 

2 

1; 


1 

+++++ 

1 

58 

Compare  posit  to  all  HVAAs 
4 
1; 

Competency:  Distance  calculation  from  data  or  screen  view,  Compulation  skills,  visual  comparison 


1 

59 

4 

4 

1 

1 

1 


1 


1 

1 

59 

Determine  nearest  HVAA 

4 

1, 

Competency:  Situational  awareness,  Planning 
1 

63 

5 
4 
1 
1 
1 


1 


2 

1; 


1 

+++++ 


A-30 


1 

60 

Monitor 

4 

1; 

Competency:  Situational  awareness 
1 

82 

4 

6 

1 

1 

1 


1 


2 

1; 


I 

1 

61 

Deicminc  friendly  non-HVAA 
4 
1, 

CompcicncN :  Categorization,  Symbology  knowledge 
! 

8: 

6 

1 

1 

1 


1 


1 

I 

('hcek  II\  AA  fuel  level 
4 
1. 

CompetencN  Knowledge  of  heuristic.  Situational  awareness 


111 

110 

6 

4 

1 


A-31 


1 

1 

1 

4 

1; 

1; 


1 

t  M  t-H 
1 

64 

Dispatch  HVAA 
4 
1; 

Competency:  Decision-making,  system  use 

Observ^abie:  HVAA  commit  action  and  target  commit  action;  verbal  comm  with  pilot 

0 

9 

2 

1 

1 

I 


1 


1 


1 

1 

82 

Assess  for  support 

4  ■  . 

1; 

Competency:  Situational  awareness 


(>() 

57 

3 

C 

1 

1 

1 


A-32 


1 


1 

90 

Hostility  level? 

4 

1; 

Competency:  Symbol  recognition,  categorization,  and  global  awareness 

4 

54 

55 
61 
100 
1 

3 

1 

1 

1 


1 


4 

1, 

1; 

1; 

1; 


iag.=lag-^K 


1 

1 

100 

Deictmmc  HVAA 
4 

Compcienc\  C'aicgonzaiion,  Symbology  knowledge 
1 

53 


1 

I 

1 


1 


1. 


A-33 


-H-H-+ 

1 

101 

Radar  guided  armament 

4 
1; 

Competency:  Knowledge  of  assets 
1 

104 

5 
1 
1 
1 
1 


1 


2 

1; 


1 

I 

102 

Hcat'Seeking  armament' 

4 

1. 

Competency;  Knowledge  of  assets 

1  ' 

103 


1 

1 

1 


1 


i 

103 

Send  hi>nx’ 

4 

1. 

Competency;  Situational  awareness 

Observable  C’all  button  engage,  pilol/flight  changes  course  to  base 

0 

6 

1 


A-34 


1 

1 


1 


1 


4 

1; 

1, 


1 

1 

105 

rucl  okay 

4 

I; 

C'ompcicncN  C'orrecl  use  of  heuristic 
obscr\  able  length  of  time  since  last  refueling 
1 

10* 

0 

1 

1 

1 


A-35 


1 


f++++ 

1 

106 

Fuel  low 
4 
1; 

Competency:  Correct  use  of  heuristic 
observable:  length  of  time  since  last  refueling 
1 

108 

7 

1 

1 

1 

1 


1 


2 

1 


1 

107 

Okay  to  commit 
4 


Competency:  Decision-making 

Observable:  AC  slays  in  pattern 

0 

8 

0 

1 

1 


1 


1 


1 

108 

Send  for  re-fuel  I ng 
4 
1; 

Competency:  Decision-making 

observable:  comm  between  WD  and  pilot,  change  in  AC  heading,  toward  tanker,  call-in  to  tanker  WD 


A-36 


0 

8 

1 

1 

1 

1 


1 


1 


1 

++-H-+ 

1 

109 

Commit  to  target 

4 

1; 

Competency:  Decision-making 
1 

64 

5 
”» 

1 

1 

1 


1 


1 

1 

110 

f  uel  too  lo\\ 

4 

1. 

(  ompctcnc)  (  orrccl  use  of  heuristic 
obNcr\ahlc  length  oriimc  since  last  refueling 
1 

ii: 


4 

1 

I 

1 


I 


1; 


A-37 


1 


1 

in 

Fuel  okay 
4 
1; 

Competency:  Correct  use  of  heuristic 
observable:  length  of  time  since  last  refueling 
1 

109 

7 

2 

1 

1 

1 


2 

1; 


1 

++++-*- 

1 

112 

Dcicrmine  next  nearest  HVAA 
4 
1: 

Competency:  Decision-making,  situational  awareness 


1 

63 

8 

4 

1 

1  ■ 
1 


1 

++++-»- 
1  ■ 

115 

Track  in  system 
4 
1; 


A-38 


1 

90 

0 

3 

1 

1 

1 


1 


2 

1; 


1 


! 

20 

Determine  mission  type 
0 
1; 


Competency:  Understanding  the  overall  mission 


3 

-) 

3 

4 
1 
1 
1 
1 
1 


1 


4 

missic>n_t>pc==l ; 
mission_t\pc==2; 
mJssion_i>pc==3; 


1 

1 

no 

Start 

n 

1, 


1 

20 

0 

1 

1 

1 

1 


A-39 


1 


2 

1; 


1 

+-HH-+ 

1 

400 

Periodic  track  update 
0 
1; 


1 

400 

0 

2 

1. 

I 

1 


1 

0.2;  1 12  seconds  -  same  as  radar  sweep  of  an  AW  ACS} 


1. 


count  =1 : 

while  count  <=  tot_aircraft  do 
current  =  aircraft(countl. 
IPDATiLPOS. 
count*^=  ] , 


5(hi 

Trace  Data  (ialhcring 
o 
1. 


1 


1 


1 


A-40 


I  {■  ■■H-+ 

41 

1 

7 

223 

1 


-H-+4-+ 

1 

21 

21 

1 


+++++ 

1 

71 

90 


1 

87 

252 

1 


1 

114 

200 

3 

acjd[Ug|  <  9999; 


1 

123 

201 

3 

ac  kiltaiil  '1 21 . 


1 

124 

211 

3 

((ac_Klliag)==21 1  1  ac_id[tag]==4l5)  \  ac_id[tag]==213); 


++-H“+ 

1 

125 


A-41 


220 

1 


++-1-1-+ 

1 

221 

221 

1 


+++++ 

acjd 

AC  identification  number  for  each  AC  by  tag 
3 
1 

500  / 

0.000000 


ackn 

Acknowledgement  of  request  given 
I 

0.000000 

-K+++-J- 

adding 

Restricts  addition  of  new  tracks 
1 

0.000000 

+++-!-»- 

airjimc 

keep  total  time  HVAA  has  been  in  air 

4 

I 

300 

0.000000 
>  aircraft 

keeps  track  of  aircraft  tags 

3 
1 

50 

0,000000 

Altitude 

AC  altitude  in  feet  by  tag 

4  ■ 

1 

500 

OOIHKKMI 

altitudein 

altitude  of  a  ne\^  track 
1 

000(K)0O 

amplin 

1 

0.000000 

1-++++ 

amplop 

1 

0.000000 

+++++ 


A-42 


arr_mod_hook 

2 

0.000000 

arr_mod_id 

Latency  for  identificatio  by  model 
2 

0.000000 

-h-H-H- 

arT_op_hook 

2 

0.000000  . 

arr_op_id 

Latency  between  arrival  an  optr  ident 
2 

0.000000 

+++++ 

arrival 

Time  a  new  track  arrived 
2 

0.000000 

atime 

Time  of  request  acknowledgement 
2 

0,000000 

attacking 

track  attacking  hostilcs 

3 

I 

300 

0  (KK)OOO 

avail_H\’.*\A 
[|\'A.-\  resource  counter 
1 

O.OtXlOOO 

beann 


0.000000 

Hearing 

beanng  of  track 
4 
1 

25.0 

0  (  ;K  H  M  H  M  ) 


bearing  1 

bearing  trom  one  .A(’  to  another 
*) 

0  OOiKiOo 
bcanng2 

beanng  Irom  one  .AC  to  another 
0 (KMKKH) 
bearop 
1 

0.000000 

bracket! 


A-43 


one  edge  of  a  range  of  bearings 

2 

0.000000 

bracket2 

one  edge  of  a  range  of  bearings 
2 

0.000000 

+ 1  4'++ 

cie_ann 

CIE  value  for  armament  of  HVAA  chosen  by 
1 

0.000000 

+++++ 

cie_fue12 

CIE  value  for  fuel  check  2  type 
1 

0.000000 


cie_hvaa 

CIE  value  for  hvaa  type  for  CIE  output  file 
1 

0.000000 

++-HH- 

cie_vecior 

CIE  value  for  vector  chosen  by  op 
1 

0.000000 

clock 

System  Variable 
2 

0.000000 

+++++ 

cnOl 

i 

0.000000 

cnnoin 

1 

0.000000 

com_count 

Communications  counter 
1 

0.000000 

+++++ 

comhostop 

Squad  p  of  hostile  committed  by  operator 
3 
1 

50 

OOOOOtK) 

comhostile 

Squad  num  of  hosule  that  was  committed 
1 

0.000000 
++-M'  + 

Com_Hvaa_Op 

Squad  P  of  committed  hvaa  by  operator 
3 
1 

50 

0.000000 

+++++ 

com_hvaal 

Squad  number  of  the  committed  hvaa 


1 

0.000000 

-H-+++ 

com_hvaa2 

Squad  number  of  the  committed  hvaa 
1 

0.000000 
M  t++  ' 
com_time 

the  time  of  the  committ 
2 

0.000000 

com_time_op 

the  time  of  committ  by  the  operator 
4 
1 

50 

0.000000 

++-HH- 

commit_count 
count  the  commits 
1 

0.000000 

+++++ 

comp_I 

the  first  competency  listed  for  a  task 
1 

0.000000 

comp_2 

the  second  competency  listed  for  a  task 
I 

0.000000 

comp_3 

the  third  competency  listed  for  a  task 
1 

0.000000 

corr 

C'orrelation 

1 

0. 000000 
count 

generic  counter 
1 

O.OiKKKH) 

C'oursc 

4 

1 

5(H) 

0  (MKHHM) 


course  known 

3 

1 

250 

0  000000 
coursein 
2 

0.000000 

+++++ 


courseop 


2 

0.000000 

■I  I  t  i  + 

CPA_range 

range  of  hostile  ship  CPA 
4 
1 

300 

0.000000 

-H-+++ 

ctype 

Communication  type 
1 

0.000000 

++-H-+ 

current 

Curent  tag  of  interest 
1 

0.000000 

dbugon 

1 

0.000000 

dclia_ali 

almudc  della  between  two  aircraft 

0. 000000 

dneiin 

j 

OOOOOOO 

do_comparc 

llag  for  comparing  c,\p  vs.  oper  actions 
1 

0  000000 
drop 


0  0(K)0rXJ 
dsnsm 


0  (HWMXK) 
dMisop 

1 

O  OmMMHl 

duration" 

\  anabie 

(KMMHHH) 

emu 

I  milter  reqort  sent 
1 

0  ()0(K)00 
l-.mit_rcq 

request  for  emitter  on  a  track 

3 

I 
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300 

0.000000 

I  I  f ++ 

Emit_stat 

Is  emitter  correlated  with  track 
3 
1 

250 

0.000000 

■>  I  M  + 

Emit_typ 

Type  of  emitter  associated  with  track 
3 
1 

300 

0.000000 

+++++ 

emitreq 

Emitter  request  sent 
1 

0.000000 

++-M-4- 

emitsiai 

1 

0.000000 

engage 

variable  to  track  whether  or  not  to  engage 
3 
1 

300 

0.000000 

escort 

3 

1 

300 

0.000000 

ESM 

3  . 

1 

250 

0.000000 

liSMkmnui 


300 


0 000000 
esmset 

to  keep  track  of  model  selling  esm  info 
3 
1 

300 

O.OOOOOO 

cstatm 

fncndly{  1 }.  commercial  (2)or  hostile  (3)csm 
1 

0.000000 

++'+"(-+ 

ctyp 
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1 

0.000000 
t  )  I  4  + 
etypin 

2 

0.000000 

fmalp 

final  priority 
1 

0.000000 
1111  + 
foundjrack 
Track  found  in  search 
1 

0.000000 

+++++ 

Friendly 

keeps  which  tags  are  friendly  AC 
3 
1 

100 

0.000000 

+++++ 

from 

array  index  for  x,y  position 

1 

0.000000 

++-M-+ 

fuel_lcvcl 

estimated  fuel  level  of  HVAA 
3 
1 

500 

0.000000 

+++++ 

fucl_ok 

Hag  for  fuel  assessment  routing  condition 

1 

O.OOOOOt) 

+  +  4-+-!- 

hookcount 

3 

1 

10 

0.000000 

hooktimc 

Time  a  hook  was  made 
0  OnOOOf) 
host  count 

counts  hov\  many  hosiilcs  are  attended  to 

1 

0  OOOOOO 
Hosttracks 

Number  of  IIV'A.A  tracks 

1 

0.000000 

-+*++++ 

host_typc 

set  true  hostility  of  AC  in  scenario 
I 

0.000000 

+4-M-+ 

Host_type 


A-48 


set  true  hostility  of  AC  in  scenario 
3 
1 

100 

0.000000 

-HH-++ 

hostile_act 

flag  for  hostile  act  by  an  AC 
3 
1 

500 

0.000000 

-H-H-+ 

HVAA 

keeps  which  tags  are  HVAA 
3 
1 

50 

0.000000 

++-H-+ 

hvaa_op 

the  hvaa  chosen  by  the  operator 
1 

0.000000 

+4-M-+  ■ 

HVAAjracks 
Number  of  HVAA  tracks 
1 

oooooon 


1 

OOOOOOO 

ical 

«  ol'dimens  on  \\h\ch  tracks  are  compared 
1 

oriooooo 

j_cai2 

s  d I  mens  to  check  for  info 
1 

0  fKKK)()0 
ID 

ID  number  for  AC'  in  scenario  events 
1 

(I  ononfMi 


id  count 

Counter  used  in  identification 

I 

O  0(  H  H  M'  X  I 


Idem  I  lied 

1  ime  track  v^a>  identified 
4 

! 

2  5o 

{)  0( H)( 

IDff 

idetntf)  friend  or  foe 

3 
1 

250 

0.000000 

idffin 
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2 

0.000000 

1  nt  + 

idffop 

1 

0.000000 

IDplat 

2 

0.000000 

++-H-+ 

IDplatf 

Type  of  platform 
3 
1 

250 

0.000000 

++-H-+ 

idplatin 

type  of  platform  -  new  track  in 
1 

0.000000 

idplatop 

1 

0.000000 

idtypin 


0.000000 

idi>'pop 

1 

0.000000 

!Ff*_known 

3 

2 

300 

5 

0.000000 

lff_modc 

Mode  of  trandponder 

3 

I 

250 

0  0(KM)(K) 

jffscl 

3 

1 

300 

0.000000 

iffmodm 


0.000000 

+++++ 

iffmodop 
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1 

0.000000 
-!■  +  +-{-+ 
iffp 

1 

0.000000 

index 

index  to  operator  data  arrays 

1 

0.000000 

+++++ 

keyin 

Input  buffer  for  keypress 

1 

0.000000 

++-H-+ 

Last_Upd 

Last  time  position  was  updated 
4 
1 

500 

0.000000 

+++++ 

maxa 

1 

0000000 

Maxspd 

Array  ot'  max  speeds 
3 
1 

250 

0  000000 
minrange 

Use  to  keep  the  min  range  to  a  target 

2 

ooooooo 

mina 

1 

0  (KXXKK) 

mmin  .  ^  ' 

minutes  in  to  convert  lime 
I 

0  orWKKX) 


miss  ty  i 

Hag  lor  missile  i>pe  radar 
1 

0  OtXKHXi 
miss  l\  2 

tlag  for  missile  typic  heal  scaker 
i  ^ 

0  0(KHX)O 
Miss_l>pcl 

Hag  tor  radar  missile  for  each  AC  by  tag 
3 
1 

500 

0.000000 

Miss_type2 
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flag  for  heat  seak  missile  for  each  AC  by  tag 
3 
1 

500 

0.000000 


mission_type 
set  type  of  mission 
1 

0.000000 


mod  hook 


0.000000 

+-1-1-++ 

mspd 

max  speed  in 
1 

0.000000 


number  of  positions  in  pattern  flight 


0.000000 


tag  value  for  HVAA  nearest  target 


ncxt_poini 

next  point  counter  m  pattern  flight 
3 

I 

500  3 

0  000000 

no^emilter 

No  emitter  report  given 


not_done 

generic  flag  for  logic  statements 


not  fuell 

flag  tor  not  enough  fuel  in  hvaa  chosen  by  op 


m^t  hostile 

bvH>le3n  lor  hostile  action  flag 


flag  for  op  H\'.-\.‘\  that  is  not  nearest 


Num_AssuMos 

track  number  of  assumed  hostile  or  unk  assu 
1 

0.000000 


Num_rncnd 

track  number  of  friendlies 


1 

0.000000 
+-H-H- 
Num_Hostile 
track  number  of  hostile 
1 

0.000000 

++-1-++ 

Num_HVAA 

keep  count  of  number  of  HVAAs 
1 

0.000000 

+-H-H- 

num_tracks 

number  of  lacks  in  the  scenrio 
1 

0.000000 

++-H-+ 

numfixes 

number  of  eye  fixations  during  search 
1 

0.000000 

++-H-+ 

objective 
System  Variable 
2 

0.000000 

+++++ 

occup 

status  of  any  AC 

3 
1 

500 

0.000000 

OCl-rom 

Operator  communcation  from 

4 
1 

500 

0,000000 

OC'Rcq 

('ommuncation  requested  by  another  oprtr 

3 
1 

1000 

0.000000 

OCTimc 

Operator  communcations  time 

4 
1 

5(Mi 

(jOOOOOO 

OClo 

Operator  communcation  TO 
1 

500 

0,000000 

OCTrack 

Track  OPl:R.\TOR  refers  to  in  communication 
3 

1  , 

1000 

0.000000 


OCType 

Operator  communcation  type 
3 
1 

500 

0.000000 

++-H-+ 

OKey 

Keypress  entered  by  an  operator 
3 
2 
5 

1000 

0.000000 

+++++ 

Ondeck 

is  ship  based  A/C  on  deck? 

3 

1 

200 

0.000000 

++-H-+ 

ondkin 

is  a/c  on  deck? 

1 

0.000000 

++++•+ 

op_hook 

2 

0.000000 

+++++ 

op_id 

Time  operator  id'd  a  track 
2 

0.000000 

op_mod_hook 

0.000000 

op_mod_id 

Time  relative  to  model  tor  operator  id 
2 

0.000000 

+++++ 

ops_avail 

Number  of  operators  of  a  type  available 
3 
1 

10 

OOOOOtK) 

optr 

Operator  a.ssicncd  to  a  ta.sk 
2 

0.000000 

oplrm 

Input  buffer  for  operator  number 
1 

O.OOOOOU 

optrout 

Operator  sending  a  message 
1 

0.000000 

++-4-++ 

optrrec 


Operator  recieveing  a  message 
1 

0.000000 

1  \  i-++ 

OReq 

Operator  request  for  the  track 
3 
1 

10000 

0.000000 

-HH-H- 

Origin 

Identify  country  of  origin 

3 
1 

250 

0.000000 

+++++ 

originin 

2 

0.000000 

OTime 

Time  an  operator  action  occurred 

4 


1000 

0.000000 

OTrack 

Track  number  operator  action  is  on 


1000 

OOOOOOo 

oisicp 

Time  step  for  actual  operator  data 


0  oodooo 

pattern 

boolean  tlag  if  AC  is  Hying  a  pattern 
1 

0  OtHXMin 
Pattern 

paucni  llag  for  each  A('  by  tag 
1 

0  (MMKMN  » 

Postlion> 

number  o\  positions  for  pattern  flight  by  lag 
3 
1 

500 

O.OOOOOO 
pre\  point 

previous  point  counter  m  pattern  flight 
1 

500 
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0.000000 

HIM 

prioritytest 

snapshot  test  variable-1 5 
2 

0.000000 

processing 

Tag  currently  being  processed 
3 
1 

250 

0.000000  ' 

\++++ 

quadin 


1 

0.000000 

quadop 

1 

0.000000 

+++++ 

Quadrant 

quadrant  of  arrival  for  track 

3 
1 

250 

0-000000 

++-(-+'+ 

Range 

Range  from  the  ship 

4 
1 

250 

0.000000 

+++++ 

range 

range  between  two  aircraft  in  three  dimensio 

*7 

O.OOOOOt) 

rangein 


0.000000 

+ 

rangeoop 

I 

0  OOOOOO 
requested 

Communication  was/ was  not  requested 
I 

0.000000 

ROi*: 


0.000000 

run 

System  \'anable 
1 

0.000000 

secsin 

seconds  in  to  convert  time 


1 

0,000000 

•}■++++ 

seed 

System  Variable 

1 

1,000000 

HIM 

sin 

1 

0.000000 

+++++ 

Speed 

Speed  of  track 
4 
1  , 

500 

0.000000 

+-I-H-+ 

speedin 

2 

0.000000 

spccdop 


0.000000 

++»-►-*- 

Squad_couni 

Number  of  AC  in  squad  by  squad  id  num 
3 
1 

10000 

OOOOOOO  * 

Squad_ld 

Squad  ids  for  each  aircraft  by  tag 
3 
1 

500 

0  oooooo 

squadnum 

number  of  A('  in  squad 
1 

0  OOOOfK)  - .  ' 

squadin 

squad  id  number 
i 

0  OOOOOO 

Slat 

1 

0  (KHKMMJ 

Statin 

*> 

OO(KK)Or) 

statop 


OOOOOOO 

statpop 
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1 

0.000000 

1  ■>  +++ 
tjo_point 

time  to  fly  to  next  point 

2 

0.000000 

++4-++ 

tag 

System  Variable 
1 

0.000000 

+++++ 

tagin 

Input  buffer  for  tag  numbers 
1 

0.000000 

++-H-+ 

task^num 

the  number  of  a  task 
1 

0.000000 

+++++ 

temp 

tcmporar>’  storage  variable 

T 

0.000000 

tempi 

iemponir\‘  integer  variable 
1 

0  000000 
temp  2 

temporary-  integer  variable 
1 

0  000000 
test  I  Off 


0 (KKKHX) 
lestlDtypc 


0  oOfHKXi 
testkl 


1)  I  UK  M  1(10 


Ih 

(  iir.Ntar.i  tu  iJeniit\  I IC 

OOlHKiOo 

IK’  riK* 


0(.HKX)()0 

Timejn 

Time  of  track  arrual 
4 
1 

500 

0.000000 


time_step 

time  for  updating  of  current  flight  interval 

2 

0.000000 

++-H-+ 

timein 

Input  buffer  for  times 
2 

0.000000 

+-H-H- 

to 

array  index  for  xy  position 
1 

0.000000 

+-HK-+ 

tot_aircraft 

counts  the  total  aircraft  in  air  space 
1 

0.000000 

+++++ 

tol_friendly 

counts  the  total  friendly  AC  in  air  space 
1 

0.000000 

tot_HVAA 

counts  number  of  HVAA  in  airspace 
1 

0.000000 

total_com 

Total  number  of  OPERATOR  communcations 
1 

0.000000 

total  _curren 
Total  current 
1 

0.000000 

tota]_op 

Total  number  of  operator  actions  for  an  optr. 

3 

1 

5 

0000000 

track 

1 

0,00{)000 

Track  H\’.'\An 
track 
3 
1 

30(1 

0.000000 

track_id 
Track  identifier 
1 

0.000000 

Track_Num 

Track  number  associated  with  tag 
3 
1 

250 
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0.000000 

HIM 

trackin 

1 

0.000000 
>  f++-t- 
trackinl 
IDS  trackin 
2 

0.000000 
+  f-H-+ 
trackin2 

1 

0.000000 

trackin3 

I 

0.000000 

++-H-+ 

tracknum 

track  num  from  array 
1 

0.000000 

trackop 

I 

0.000000 

4"4-f-++ 

tracks 

1 

0.000000 

irackiesi 

snapshot  debug  variable 
1 

0.000000 

trackioi 

2 

0.000000 

updt 


0.000000 

updiop 


OOOfJOOO 

user 

I 

0.000000 

vector 

pursuit=  I ,  nose=2,  sidc=3 
1 

0.000000 

+++++ 

Vector_Op 

pursuit=l ,  nose-2,  side=3  by  operator 
3 


1 

50 

0.000000 

watch 

number  of  watchstations 
1 

0.000000 
+++++ 
watch  tot 

total  num  watch  stations  added  in  scenario 
1 

0.000000 

+-H-I-+ 

WD 

Constant  identifier  for  WD 
1 

1.000000 

WD_prio 

WD  priority  for  final  threat  determination 
3 
1 

300 

0.000000 

WD_req 

} 

1 

300 

000000(1 

Weapons 

Array  -  number  of  weapon  types 
3 
1 

250 

0  0000(0 
\\eapi»n>  l\ 

1 

0  00000(1 
wpnsm 

Number  of  capons  type  for  a  new  track 
1 

0  (WKKKMI 


\  eoi>rdinale 

T 

0  00O0( Ml 
\ 

\-Ci>ordmaie  ol  tae 

4  '■ 

I 

>{  HI 

0  0000(H) 

XAWAC  S 

x-coordinates  of  .A  W.ACS 
*)  . 

0.000000 

xHVAA 

X'Coord males  of  currently  selected  HVAA 
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0.000000 


yop 


0.000000 

Ypos 

Y  coord  of  lag  and  position  n 
4 
2 

500 

50 

0.000000 

++4-+-1- 

zoff 
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1 

0.000000 
\  \  \  \-+ 
zpos 

2 

0.000000 

ALT_UPDATE 

Updates  AC  Alt  from  scenario  event 
Aititude[tagin]  :=altitudein; 

squadin:=(-9999); 

altitudein:=(-9999); 


CHECK_COURSE 

Has  next  point  been  reached  in  pattern  flight 
time_siep  ;=  clock  -  Last_Upd [current];  {in  minutes} 
n  :=  next_poini[currenl];  {next  course  change  position} 

tjo_point  :=  distance(X[current],  Y[curTent],  Xposfcurrent,  n],  Ypos[current,  n])  /  (Speed [current]  /  60); 

if  t_io_poini  <  time_step 

then  Lasi_Upd[curTent]+-t_toj5oint, 

REPOSITION, 

UPDATE_CRS, 

l,ast_Upd[current]-={time_step-t_to_point), 

REPOSITION 

else 

REPOSITION; 


CIi:_t)ATA 

Sets  up  the  data  for  the  CIE  outpuf  file 

ifnoi  ncarcst  ==  0  &  not_fueI  1  ==  0 

then  cic_h\aa  •=  1 ; 

if  not  nearest  ==  0  &  not_fucIl  =-  ] 

then  cie  hvaa  -2, 

if  noi  nearesi  ==  1  &  not_fucn  ==  0 

then  cie  hvaa  -  3, 

if  noinearest  ==  1  &not_fuell  =  1 

then  cie  hvaa  4; 

if  fucl  ok  then  cic_fucl2  =l  else  cie_fuel2:=0; 

if  Miss_  lypel  {nearcstl  then  cie_aim:=2; 

if  MNs_type2(nearestl  then  cie_arm;=l; 

if  MI^^_typcl {nearest]  &  Miss_typc2[nearest]  then  cic_arm:=3; 

if  Mtss_t>pel  [nearest i==0  &  Miss_lype2[nearest]==0  then  cic_ann;=4; 

cie  \cctiir  -  \'cctor_()p| index);  4 


CT.ASS_MAT(I1 
time  to  classify  item 
.00045  60. 

commit_c'oi;nt 

Count  the  number  of  commits 

!if  OKey[\VI),i]==l  then  WO_commit+=l ;} 
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COMP^HVAA 

Compare  HVAA  committed  by  exp  vs  sim  op 
hvaa_op  :=  FIND_SQUAD; 

if  hvaa_op  o  nearest 
then  nearest  :=  hvaa_op, 
not_nearest:=l ; 


I  I  M  + 

CONVERT^TIME 

convert  to  decimal  minutes 

timein:=((minin*60)+secsin)/60; 

+++++ 

DECIDE 

lime  to  make  decision 
.070/60; 


DET_HOST 

Determine  if  act  is  hostile  toward  friend 

{ Hostile  act  is  currently  based  on  distance.  Any  AC  not  friendly  within  150  of  any  friendly  is  considered  hostile} 

current  :=  FriendIy[count]; 

tf  occup(current]  ==  0  &  GET_RANGE  <  200 
then  noi_hosiile:=0,  hostiIe_act[tag]  :=  1; 

{checks  of  hostile  heading  *HOST_HEAD’  and  inside  the  adid  zone  ’IN_ADID_ZONE’  are  also  available} 


di-:t_ni:arest 

i-ind  the  nearest  MVA.A  to  target  aircraft 

count =1. 

ncarest=(). 

min_range=0. 

vv  hile  count  <-  tot_HVAA  do 

D!n_orcu[>, 

count*=l. 


Din  o( 'CUP 

Determine  if  H\‘AA  is  already  occupied 
current  =  HVAA(count); 

if  tK’cupj current]  ==  0  then 
Dl  I.  RANGE . 


Dl  I  RANG! 

Determine  the  range  from  H\'AA  to  target 

\>  distance  =  distancclXjtag],  Y(iag],  XfcuiTenl],  Yfeurrent]);  {this  is  the  two  dimensional  distance  between  the  aircraft  in  miles} 
range  ^  \\  distance. 

delta  all  -  (absolutet.Mtitudejiag]  -  AltitudefcurrentD)  /  5280;  {altitude  delta  converted  to  miles} 

{range  ■=  ((xy_distance^2)  +  (dclta_ail^2))  ^  (1/2);} 

{range  is  a  squared  plus  b  squared  equals  c  squared  where  a  is  xy  distance,  b  is  altitude  difference,  and  c  is  range} 
if  range  <  mm  range  |  min_rangc==0  then  min_range  :=  range,  nearest  ;=  current; 


DISP.MCH 
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Dispatch  HVAA(s)  to  hostile  AC 
Squad_count[Squad_Id [nearest]]  -=  2; 

if  Squad_count[Squad_Id[nearest]]  <  2  then  occup [nearest] :=1,  avail_HVAA-=l; 

+++++ 

EYE_FIX 

time  spent  in  eye  fixation 
.250/60; 

M  M-f 

EYE_MOVE 

time  spent  moving  eyes 

.170/60; 

++-H-+ 

FIND^SQUAD 

find  the  tag  of  the  squad  comitted  by  sim  op 
not_done  :=  1 ; 
count  :=1; 

while  not_done  do 

if  Com_Hvaa_Op[commit_count+l]  ==  Squad_Id[count] 
then  not_done~0 
else  count+=l; 

count; 


FUEL_CHECK1 

Check  if  HVAA  has  enough  fuel  for  mission 

{Fuel  determination  is  based  on  the  time  spent  in  the  air  or  time  since  last  refueling} 

if  (clock  -  Timc_In[nearest])  <  30  then  fuel_ok:=l 
else  fuel_ok:=0; 

if  do_comparc  &  fucl_ok==0 
then  not_fucll  :=1,  fucl_ok:=l; 


FUEL_CHECK2 

Check  if  HV'AA  has  fuel  vector  type 

{Fuel  dcieimination  is  based  on  the  time  spent  in  the  air  or  time  since  last  refueling} 

if  (clock  -  Timc_ln[nearest])  <  15 
then  fucI_ok:=l  else  fucl_ok:=Cl; 


GET^BEARING 

Ciet  the  beanng  from  on  AC  to  another 
dircction(X[from].  Y(from|,  X[lo],  Y(to]); 


gi:t_rang! 

fjct  the  range  between  two  AC 

disianccf  X(curreni],  Yfeurrent],  Xfiag],  Y(tag]); 

{current  is  the  friendly  being  checked  against  the  hostile  'tag'} 


HOST_HEAD 

Determine  if  heading  is  hostile 

{Heading  is  hostile  if  it  is  towards  a  friendly  within  two  degrees} 

from  :=  tag;  to  current; 
bcaringl  ;=GET_BEARING; 

from  ;=  current;  to  :=  tag; 
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bearing2  :=  GET_B EARING; 


i  f  bearing2+ 1 80  >  3 60  then 
bearing2  :==  beanng2  +  180-  360; 

if  bearing  1+2  >  360  then  bracket  1  :=  bearing 1+2-360; 
if  bearingl-2  <  0  then  bracket2  :=  bearing  1-2+360; 

if  bearing2  >  bracket  1  |  bearing2  <  bracket2 
then  0  else  1 ;  ' 


HOSTILE_ACT 

Check  if  AC  is  making  a  hostile  move 

{If  turned  towards  friendly  and  within  specified  range  or  within  Air  Defense  ID  Zone} 

count:=0; 

not_hostiIe:=l; 

while  not_hostile  &  count  <=  tot_friendly  do 
DET_HOST, 
couni+=l; 


++-H-+ 

ID_UPDATE 

Updates  the  AC  ID  based  on  scenario  event 
acjd(tagin]  :=  ID; 

squadin:=(-9999); 

ID;=(-9999); 


+++++ 

IN_ADID_ZONE 

Check  if  AC  is  in  Air  Defense  ID  Zone 
Xftag);  y:=Y[tag]; 

if  y>300  i  .x<(-120)  t  y<(-100)  |  x>l80 

then  0 

else  if(v-(-1.3333*x))<  (-100)! 
ty-(-2*x))<  540  1 
(y-(1.125*x))<(-l67.5) 
then  0  i 
else  1 ; 


LISTEN 

time  to  receive  acknowledgement 
3. 2 '60. 


move;_cur_t 

time  to  move  cursor  w/T-ball 
{assume  approx.  6  cm  ave  dist  to  move) 
(.3322*{log((6  .25)+,5)))/60, 


MOVi:_HAND 
lime  to  move  hand 

{assume  dtstance=l  2cm,  size=.25cm} 
0.5586585448239/60; 

NEW_COURSE 

Used  for  scenario  event  course  changes 
current  :=  tagin; 
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REPOSITION; 


Pattem[tagin]  :=  pattern; 

Positions[tagin]  :=.n+l; 

Xpos[tagin,  1]  ”  X[tagin]; 

Ypos[tagin,  1]  :=  Y[tagin]; 

count:=l; 

while  count  <=  n  do 
Xpos[tagin,  count+1]  :-  xin [count], 

Ypos[tagin,  count+1]  :=  yin[count], 
count->-=l; 

prevj3oint[tagin]  :=  1 ; 
next_point[tagin]  :=  2; 

Courseftagin]  :=  direction(X[tagin],  Y[tagin],  xin[l],  yin[I]); 


count”  1; 
while  count<=m  do 
xin[count]:={-9999), 
yin[count]:=(>9999), 

00001-^=1 ; 

paUem--9999; 

n:=.9999; 

NPAVTRACK 
Start  2  new  track 
•1  Start  a  new  track) 

Timc_  Injiagin  |:=timcin; 

I.asi_l'pd[tagm]:=clock; 

Squad_  ldliagin]=squadin; 
Squad_counilsquadin]:=squad_num; 

Spccd[iagin}  =spccdjn; 

Allitudeliaginl  ^aitiludein; 

Positu)ns[iaginl 
Pattern  jiaginj^paitcm; 

count  *  ! , 

\s  h lie  count  <=  n  do 
Xposjlagin.  count]  :=  xinfeouqt], 

Yposltagin.  count]  :=  yin(count], 
ci>iint  *  -  1 . 

Xltagmj  ==\in|lj:  't'ltagin]  yin[l ); 

C  oursejiagtn]  -  dircction(xin|  1  ],  yin[l  j,  xin[2],  yin|2]); 
prc\  point] tagin]  =1.  ncxl_poini|tagin]:=2; 

Mis^  i\pel  (tagin  1  =miss_ty_l ; 

^lls^  t>pe2jiagm]  -miss_ty_2; 

ac  idliagin]  “ID. 

start!  1  ItKlagin ). 

toi_aircral'f=  1 , 

aircraft] tot  aircraft]  tagin; 

(Rc-set  input  vanable  values) 

limcin=(-9999). 

squ3din={-9999); 

squad_num”(-9999); 

spcedm-(-9999); 


aUitudein:=(-9999); 

count:=l; 
while  count<=n  do 
xin  [count]  :K-9999), 
yin[cpunt]:=(-9999), 
count+=l; 

n:=(-9999); 
pattem:=(-9999); 
miss_ty_l  :=(-9999); 
miss_ty_2:-(-9999); 
ID:=(>9999); 


I  I  I  H 

PERCEIVE 
time  to  perceive 
.100/60; 

+++++ 

PUSH_BUTTON 
time  to  push  a  button 
.4/60; 


REPOSITION 
Reposition  individual  track 
I  Determine  new  position  on  x,y  grid } 

{if  Course[current]>=0  &  Course[curTent]<90  then  } 

X{current]:=  X[current]+(  Speed  [current]  ♦  ((clock  -  Ust_Upd[curTent])/60)*cosine(Course[curTent])); 
Y[currenl]:=  Y[current]+(  Speed  [current]  *  ((clock  -  Ust_Upd[curTent])/60)*sine(Course  [current])); 


{if  Courselcurrent)>=90  &  Course[curTent]<180  then 

X[currcnt]-  X[current]+{  Speed[curTent]*((clock  -  Last_Upd(curTent])/60)*cosine(Course[current]-90)), 
Y(currcni]:=  Y[currentj-(  Speed[current]*((clock  -  Last_Upd[current])/60)^sine(Course[current]-90)),  temp:==2; 

if  Coursc|currentl>=180  &  Course[curTent]<270  then 

Xlcurrentl:=  X[current]-(  Spced[curTent]*((clock  -  Last_,Upd[current])/60)*sinc{Course[curTenl]-l80)), 
Ylcurrcntli^  Y[current]-(  Spced{current]*((clock  -  Last_Upd[current])/60)*cosine(Course[current]-l  80)),  lemp:=3 

if  Coursc[currcnt)>=270  &  Course[current]<360  then 

Xlcurrcntj-  X[currentj-(  Speed[cuTTent] ♦((clock  -  Last_Upd(current])/60)*cosine(Course[current]-270)), 
Y(currcnl):=  Y[currcntj+(  Specd[curTent]^((clock  -  Last_Upd(curTent])/60)*sine(Course[current]-270)),  temp:=4;| 

Tasi_Updl  current  I  :=  clock; 


4-  +  ++  + 

SPI)_UPD..\Ti: 

Updates  the  speed  from  a  scenario  event 
Spced{tagin]  -  specdin; 

speedin=(-9999), 

squadin:=(-9999), 

SPEAK 

lime  to  issue  commands 
{assume  6.8  words/transmission} 

2/60; 

+++++ 

TRACE 

Gathers  data  for  the  expert  trace 
task_num  ;=  taskO; 
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start(500,  999); 


++-H-f 

UNNAMED39 


-l^■^+++ 

UNNAMED42 


+++++ 

UPDATE_CRS 

route  to  type  0  or  type  1  pattern  updates 
if  Pattern  [current]  ~  0  then  UPDATE_CRSO; 
if  Pattern  [current]  =  1  then  UPDATE_CRS1 ; 

+++++ 

UPDATE_CRSO 
Updates  course  for  pattern  type  0 
if  next_point[current]  “  Positions[current] 
then  Pattern  [current]  :=  3; 

if  next_point[current]  <  Positions[curTent] 
then  prev_point[curTent]  :=  next_point[cuiTent], 
next_point[curTenl]+=  I, 
n  :=  nexi_poini[current]. 

Course  [current]  :=  direction(X[curTent],  Y[current],  Xpos[current,  n],  Ypos[curTent,  n]); 


UPDATE_CRS1 

Updates  course  for  pattern  type  1 

not_donc:=l ; 

if  ncxi_point(curTeni|  -=  Posiiions[curTent]  &  Positions[current]— 2  then  Pattern  [current]  :=4, 
time_stcp  :=  0, 
t_io_pomt  ~  0; 

if  ne\t_point[curTeni]  <  Posilions[curTent] 
then  prc\  _poini|currcnt]  :=  nexij30int[currenl], 
ne\t_poini[eurrcntl+=  1. 
n  -  nc\t_pomt[cuTTcnt|, 

('oursej current]  ;=  direction(X[current),  Y[cuiTent],  Xpos[current,  n],  Ypos[curTent,  n]),  not_done:=0; 

jf  not  done  &  ncxt_point[curTeni]  ==  Posilions[current]  &  Positions[current]>2 
then  prc\_pomt[currcnt]  :=  next_point[current], 
nc\i_point[currcnt]  :=  2,  - . 
n  ■=  ncM_pointlcurrenl]. 

C'oursej current]  direction(X[curTent],  Y[current),  Xpos{current,  n],  Yposjeurrent,  n]); 


UPDAIl  POS 

Update  the  position  of  an  AC 

I  A('  uiih  pattern  t>pc  4  arc  holding  station  by  circling  a  point:  For  the  purpose  of  the  modeU  these  will  not  change  position  so  we  will  use 
Kl-POSITION  with  a  speed  of  zero} 

tf  Patient  [current  j===4 
then  temp  =Spced(curTcnt], 

Speed}  current  I  =0, 

RFPOSITION, 

Speed!  current  I :  =iemp. 


[AC  with  pattern  t>pe  3  just  do  an  update  without  any  check  for  course  change} 
if  Pattem[current]==3  then  REPOSITION; 
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{AC  with  pattern  type  0  use  next  point  for  course  until  they  have  reached  their  last  point.  They  are  then  type  3s} 

if  Pattern  [current]==0 
then  CHECK_COURSE; 

{AC  with  pattern  type  1  use  next  point  for  course  and  continue  with  a  cyclical  pattern  of  points) 

if  Pattern  [current]”  1 
then  CHECK_COURSE; 

M  M-f 

VIS_SEARCH 

time  spent  visually  searching 

(numfixes*(EYE_MOVE+EYE_FIX))/600; 

4-+4-++ 

Tlfltlt***** 

0.000000 

3 

0 

0 

1 .000000 
1 

1000.000000 

mission_t>'pc:=2;  {Transition  type} 

{Only  mission  t>'pe  2  is  available  at  this  time) 


0.000000 

} 

0 

0 

1  .oooooo 

1 

1000  000000 
;l:sc  comparison  setting} 
{0  not  using  1  =  using} 
do  comparc  :=  1 , 

OtlOOOOO 


0 
0 

1  0(K)0()f) 

1 

KKXi OOOOOO 
{.Stmulalor  Operator 
(  om  H\aa_Op[  1  j  = 
C  om  H\aa  ()p|2)  = 
(  om  H\aa  ()p{3)  = 
(  om  Ih aa  Op|4 j  = 
(  om  fl\aa_Opl5j  = 
Com  ll\a3_()p}(»j  = 
Com  H\aa  ()p{')  = 
Com  Hvaa  ()p{S|  - 
(  om  H\aa_(>p|Oj  = 
Com  H\aa  Opj  H»j 
(.  om  H\aa_{)p[  1 1  j 
(  omjhaa  ()p{  12) 
Com  lhaa  {)p[  1 3j 
(*om_Hvaa_()p[  14] 
Com_Hvaa_()p|  15] 
Com  Hvaa  Opj  16) 
Com_  I  lvaa_Op|  17) 
Com_Hvaa_(3p(  18) 
('om_Hvaa_Op[  19) 
Com_Hvaa_Op[20] 
Com_Hvaa_Op|21 ) 


Observable  Data} 

1018;  Vector_Op[l  ]:=1 
1018:  Vecior_Op[2]:=l 
1018,  Vccior_Op|3}:=l 
1018,  Vector_Opi4|:=l 
1032.  \’ccior_Op[5]:=l 
1032:  Vcctor_Op[6]:=l 
1018,  Vcctor_Op[7]:=l 
1018,  Vector_Op[8]:=l 
1032.  Vcctor_Op[5]:=l 


=1032, 

Vecior_(3p[10] 

=  1 

=1018, 

\T‘cior_Op[l  1  ] 

=  1 

=1018: 

Vector_Op[12] 

=  1 

=1018: 

Vcctor_Op(13] 

=  1 

=1018; 

V'eclor_Op[14] 

=  1 

=1018; 

Vcctor_Op[15] 

=3 

=1018, 

Vector_Op[16] 

=3; 

=1018; 

Vector_Op(17| 

=3; 

=1018. 

Vector_Op|18] 

=3: 

=1034: 

Vector_Op[  19] 

=2 

=1034; 

Vcctor_Op[20i 

=2 

=1034; 

Vccior_Opi21  j 

=2: 
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Com_Hvaa_Op[22]:=1034;  Vector^Op[22]:=2; 
Com^Hvaa_Op[23]:=1077;  Vector_Op[23]:=3 
Com_Hvaa_C)p[24]:=1077;  Vector_Op[24]:=3 
Com_Hvaa_Op[25j:=1078;  Vector_Op[25]:=2 
Com^Hvaa_Op[26]:=1078;  Vector_Op[26]:=2 
Com_Hvaa_Op[27]:=1077;  Vector_Op[27]:=l 
Com_Hvaa_Op[28]-1077;  Vector_Op[28]:=l 
Com_Hvaa_Op[29]  := 1 052 ;  Vector_Op[29]  :=  1 
Com_Hvaa_Op[30]:=1052;  Vector_Op[30]:=l 
Com_H vaa_Op [31]:=1052;  V ector_Op [3 1  ] :=  1 
Com_Hvaa_Op [32 ]  := 1 052 ;  V ector_Op [32 ] :=  1 


0.000000 

3 

0 

0 

1. 000000 
1 

1000.000000 

{Airbase  friendly  -  Jaffna  -  SLJA} 

X[151):=30;  Y[151]:=205; 

tot_friendly+=l ;  Friendly[tot_friend!y]:“151 ; 

+++-f+ 

0.000000 

3 

0 

0 

1 .000000 

1 

1000.000000 

{Airbase  friendly  -  Trincomallee  -  SLTE} 
X11521  -  65;  Y[I52]  135; 

tot_fricndly-*-=l ;  Friendly [tot^friendly]  152; 


0.000000 

3 

0 

0 

1 .000000 
1 

1000  000000 

{Airbase  fncndlv  -  Ragooner  -  SLRR} 

X[1531=  35:  Y[153f:=115; 
iot_fricndly'^=l :  Friendly[tot_;friendly]  :=  153; 


0  000000 
3 
0 
0 

I  OOfHlOO 
1 

)0(K)  000000 

|.*\irhase  friendly  -  Randy  -  SLKY} 

\[154j  =40.  Y(1541;=75; 

iot_ friend ly-=^  1 .  Friendly [toi_friendly]  :=  1 54; 


0.000000 

3 

0 

0 

1.000000 

1 

1000.000000 

{Airbase  friendly  -  Colombo  -  SLCO} 


X[155]:=(-5);  Y[155]:=55; 
tot_friendly+=l;  Friendly[tot_friendly]  :=  155; 

f  {■+++ 

0.000000 

3 

0 

0 

1.000000 

1 

1000.000000 

{Airbase  friendly  -  Galle  -  SLGE} 
X[156]:=(23);  Y[156]:=13; 
tot_friendly+=l ;  Friendly[tot_friendly]  :=  156; 

0.000000 

3 

0 

0 

1. 000000 
1 

1000.000000 

{Airbase  friendly  -  Hambantota  >  SLHA} 

X[157]  :-93;  Y[157];=28; 

tot_friendIy-^=l ;  Friendly[tot_friendly]  :=  157; 

+++++ 

0.000000 

3 

0 

0 

1.000000 

1 

1000.000000 

{Carrier} 

X[I581“180;  Y[158]:=180; 
lot_fncndly+=l ;  Friendly [tol_fnendly]  :=  158; 


0.000000 

3 

0 

0 

1 .000000 
1 

1000.000000 

{AWACSIIROU  .. 

iagm:=l;  timem:=0.00;  squadin:=214;  squad_num:=l;  speedin:=380;  altitudein:=29000;  n:=5;  pattem:=0;  xin[l]:=-60;  yin[l]:=-60;  xin[2]:=60; 
yin{21:==60;xin{31:=85;  yin[3];=120;xin[4]:=80;  yin[4]:=95;  xin[5];=10;  yin[5]:=80;  missjy_l:=0;  missjy_2:=0;  ID:=415;  NEW_TRACK; 

0.000000 

3  .  ■  .  .  . 

0 

0 

!  .00{K)00 

I  < 

1000,000000 

}DCA  M1G-29S  BAAZ  -  Hostile  -  HOOO,  HOOl } 

iagin:=30,  timein:=0.00;  squadin;=53;  squad_num:=2;  speedin:=640;  aUitudein:=35000;  pattem:=l;  n:=2;  xin[l]:=-1 10;  yin[l]:=328;  xin[2]:=“ 
110:  yin[2)  =260;  miss_ty_l  ;=1 ;  miss_ty_l  :=2;  ID:=9999;  NEW_TRACK; 


0.000000 

3 

0 

0 

1 .000000 
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1 

1000.000000 

{DCAF-16CS-FRAZER31 -FZ31,32,33,34,35,36,37,40,41,42}  . 

tagin!=13;  ti mein :=0. 000;  squadin:=1052;  squad_num:=10;  speedin:=600;  altitudein .*=32000;  pattem:=l;  n.— 2,  xin[l].— 35,  yin[l].— 15, 
xin[2):=I10;  yin[2]:=30;  miss_ty_l:=l;  miss_ty_2:=l;  ID:=9999;  NEW_TRACK; 


0.000000 

3 

0 

0 

1.000000 

1 

1000.000000 

{STRIKE  F-111 A  -LAZER30-LR30,31,32,33,34,35} 

tagin:=I7;  timein:=0.00;  squadin:=1087;  squad_num:=6;  speedin:=600;  altitudein :=32000;  pattern- 1 ;  n:=2;  xin[l]:=-5;  yin[l]:=55;  xin[2]:-l  10 
yin[2]:=30;  miss_ty_l  :=0;  miss_ty_2:=l;  ID:=1 1 1;  NEW^TRACK; 

+++++ 

0.000000 
3  ‘ 

0 

0 

1 .000000 
1 

1000  000000 
{PAlI.72-TransponJ 

tagin  =5.  i!mcin:=0.0017:  squadin:=1005;  squad_num:=I;  speedin:=500;  altitudein:=30000;  pattern :=0;  n:=2;  xin[l]:=-180;  yin[l]:=240; 
.\ui[2]— -5,  ym(2);=55;  miss_ty_l:=0;  miss_ty_2:=0;  1D:=9999;  NEW_TRACK; 


O.OIOOOO 

3 

0  . 

0 

I  O(K)OfK) 

I 

!000()0000(» 

!  Jammers  (‘anbcrra  -  Hostile  >  3  -  H522,  H523,  524 1 

tagin  ^  i  timein:=()  0083;  squadin;=107;  squad_num:=l ;  speedin:=580;  aliitudein;=28500;  pattem:=0;  n:=4;  xin{l]:=-135;  yin[I]:=245; 
\in|2|  =  <100,  yin(2|  =70;  xinl3):=-10;  yin[3j:=65;  xin[4]:=-I35;  yin(4]:=245;  missjy^l :=0;  missjy_2:=0;  ID:=9999;  NEW_TRACK; 

taizin  =135,  iimein  =0.0083;  squadin:=108;  squad_num:=l ;  speedin:=580;  aititudein:=28500;  pattem:=0;  n:=4;  xin[l]:=-135;  yin[l]:=245; 
xml2I  ---100.  yin[21:=70;  xin[3]:=-l0;  yin[3]:=65;  xin[41:=-I35;  yin(4];=245;  missjy_l;=0;  missjy_2:=0;  ID:=9999;  NEW_TRACK; 

tagin  =  1 30,  iimein  =0.0083;  squadin:=109;  squad_num:=l ;  speedin:=580;  aliiludein:=28500;  pattem:=0;  n:=4;  xin[l]:=-135;  yin[l]:=245; 
xinl2j  --100.  yinl2]:=70;  xin|3]:=-10;  yin[3]:=65;  xin[4]:=-135;  yin[4]:=245;  miss_ty_l  :=0;  miss_ty_2:=0;  ID:=9999;  NEW^TRACK; 


t)020UiKi 

o 

0 

i  i  M  M  w »  K  ! 

1 

KMMlOOlKKK)  ^ 

ISOJl'i-lllA-CrovvOl  -CWOl} 

tagin  imKnn  =0  0167;  squadm:=1007,  squad_num;=l;  specdin:=465;  altiludein:=24600;  n:=3;  pattem:=I;  xin[l]:=40;yin(l]:=75;xin[2]:=- 

60,  v!n{2|  =1  So.  xin[3 j  =-40,  yin(3]:=l55; 

miss_i>  I  -0.  miss_t>_2:=0,  10;=]  1 1 ;  NEW_TRACK; 


0  020000 
3 
0 
0 

1 .000000 
1 
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1000.000000 

{SL  Maritime  Surveillance  -  Cover  01  -  CVOl ) 

tagin:=8;  timein:=0.0167;  squadin:=1008;  squad_num:=l;  speedin:-245;  altitudein:=4000.;  pattem:=0;  i 
120;  yin[2]“60;  xm[3]:=-30;  yin[3] :=-60;  xin[4]:=-20;  yin[4]:=-50;  xin[5]:=-l  10;  yin[5]:=50;  xin[6]:=- 
xin[8];=0;  yin[8]:=-35;  xin[9]:=-57;  yin[9]:=50;  xin[10]:=-5;  yin[10]:=55;  miss_ty_l  :=0;  miss_ty_2:=0 


■}4-+++ 
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0 
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1 

1000.000000 

{Air  Defense  2  -  Hostile  -  H0552} 

tagin;=68;  timein:=0.0l67;  squadin:=72;  squad_num:=l;  speedin;=620;  altitudein:=27000;  pattem:=l; 
1 80;  yin[2]:=240;  miss_ty_l  :=1 ;  miss_ty_2:=l ;  ID:=1 1 1 ;  NEW^TRACK; 


+++++ 

0.020000 
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0 

0 

1.000000 

1 

1000.000000 

{DCA  F-15C  Wolf  51  -  WF51, 52,  53,  54,  55,  56,  57,  60,  61, 62,  63,  64,  65,  66} 

iagin:=10;  timein -0.01 67;  squadin:=1018;  squad_num:-14;  speedin:=640;  altitudein:==35000;  pattern:' 

xm[2]:-60;  yin[2]:=180;  miss_iy_l  ;=1 ;  miss_ty_2:=l ;  ID:=1 11;  NEW_TRACK; 


0.020000 

3 

0 

0  .  : 

1 ,000000 
1 

1000  000000 

{Compass  Call  EC-130H  Noisy  22  -  NY22  } 

iagin:=24;  timein:=0,01 67;  squadin:=l  1 12;  squad_num:=l ;  speedin:=276;  altiludein:=27500;  pattem:= 
xin(2):=45,  yin[2):=190;  xinf3]:^30;  yin(3]:=160;  xin[4]:=-25;  yin(4]:=}40;  miss_ty_l :=0;  missjy_2:= 


0  020000 
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0 
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lOOOOOOO(K) 

{ABCCC  111  EC’-130E  Razorback  31  -  RZ31  } 

iagin:=26;  timcin:=0.01 67;  squadin:=I  1 14;  squad_num:-l ;  speedin:=285;  altitudein;=17500;  pattem:- 
yin(2):=125;  xm[3j;=45;  yin[3]:=135;  xin[4]:=27;  yin[4|:=l29;  missjy_l  :=0;  miss_ty_2:=0;  ID;=I11; 


+++++ 

0.020000 

3 

0 

0 

1 .000000 


ri:=10;  xin[l]:=-5;  yin[l]:=55;  xin[2]:=- 
■100;  yin[6]:=50;  xin[7]:=-10;  yin[7]:=-44; 
;  ID:=111;NEW_TRACK; 


i:=2;  xin[l]:=-I35;  yin[l]:=265;  xin[2]:=- 


l;n:=2;  xin[l  ]:=65;  yin[l]:=135; 


1;  n:=4;  xin[l]:=“25;  yin[l];=140; 
==0;ID:=111;NEW_TRACK; 


1;  n:=4;  xin[l]:=93;  yin[l]:=28;  xin[2]:=5; 
NEW_TRACK; 


A-74 


1 

1000.000000 

{Gunship  AC-130A  Casper  51  -  CR51}  ^  -ic  •  rn._no.  *  m— n. 

tagin:=27;  timein:-0.0167;  squadin:=l  1 15;  squad_num:=l;  speedin -285;  altitudein:=14300;  pattern:-!;  n:-6;  xin[l]:-35;  yin[l].  98,  xin[22^0, 
yin[2]:=180;  xin[3]:=25;  yin[3]:=205;  xin[4]:=45;  yin[4]:=170;  xin[5]:=35;  yin[5]:=155;  xm[6]:=75;  yin[6];=125;  miss_ty_1  :-0;  iniss_ty_2:=0; 
ID:=!11;NEW_TRACK; 


+++++ 

0.020000 

3 

0 

0 

1.000000 


1000.000000 

{U-2RELINT  Snoopy  23 -SY26} 

tagjn:=28;  timein:=0.0167;  squadin:=l  1 16;  squad_num:=l;  speedin:=500;  altitudein:=70000;  pattem:=l;  n:=7;  xin[l]:— 240;  yin[l]:-60; 
xin[2]:=-180;  yin[2]:=240;  xin[3):--65;  yin[3]:-120;  xin[4]:=0;  yin[4]:=180;  xin[5]:=120;  yin[5]:=310;  xin[6]:=50;  yin[6]:-180;  xm[7]:-65; 
yin[7];=120;  nniss_iy_l  :=0;  miss_ty_2:=0;  ID:=1 1 1;  NEW_TRACK; 


0.020000 

3 

0 

0 

1.000000 

1 

1000,000000 

}  DC  A  F-I()Cs  -  SOLAR  -  SRI  K  12,  13,  14,  15. 16} 

iaein:=l  1;  timcin;=0.01 7;  squadin:=l032;  squad_num:=6;  speedin:=600;  altitudein:=32000;  pattem:-0;  n:-6;  xin[l]:-23;  yin[l]:-13; 

xin[2|-100,yin[2]:=120;  xin[3]:=-10;  yin[3]:=l80;  xin[4I:=-25;  yin[4]:=240;  xin[5]:=37;  yin[5]:=125;  xin[6]:=23;  ym[6]:=13;  miss„ty_l:=l; 
miss_i\_2:=  1 ;  1D:=1 1 1 ;  NEW_TRACK; 


0  02000f) 

3  . 

0 

0 

1  0()(M){K> 

! 

10{)()00(KMK) 

{M!G-21  Air  Dclcnsc  1-.7M  1  -  Hostile  -  2  -  H050,  H054} 

tagin  ^4S,  iimcin:^0.0167;  squadin:=62;  squad_num:=l;  speedin:=550;  altitudcin:=29000;  pattem:=l;  n:=2;  xin[l]:=-48;  yin[ll:-270;  xin[2]:— 
25,ym|2)  ”240.  mis.s_ty_l  :=1 ;  miss_ty_2;=l ;  ID:=9999;  NEW_TRACK; 

tactn  -52.  iimcin  =  0  01 67,  squadin:=64;  squad_nurn:-l;  speedjn:=550;  altitudein;=29000;  pattem:=l ;  n:=2;  xin[l]:=-48;  yin[l):=270;  xin[2]:=- 
25,  >  m| 2  j  240.  miss_i\_  1 1 ;  miss jy_2:=  1 ;  ID:=  1 1 1 ;  NEW_TRACK, 


0  02001 M) 

3 

0 

0 

1  ()000<K) 

1 

)(K)00(MK)(y) 

riankcr  K('-!()  EXXON  40  -  EN40  4i ,  42 1 

lagin  =29;  timcm:=0.01 67;  squadin:=l  1 17;  squad_num:=3;  speedin:=400;  altitudein:=35000;  pattem:=I ;  n:=3;  xin[l):=-28;  yin[l]:=l  15; 
xin[2j:=>10.  yin[2):=145;  xin[3]:=-20;  yin[3]:=90;  miss_ty_l :=0;  missjy_2;=0;  ID:=1 1 1 ;  NEW_TRACK; 


0.020000 


A-75 


3 

0 

0 

1.000000 

1 

1000.000000 

{Rivet  Joint  CYPHER  1 1  -  CRl  1 } 

tagin:=23;  timein:-0.0167;  squadin:-!  1 10;  squad_num:=l;  speedin:=400;  altitudein:=31000;  pattem:=l;  n:=4;  xin[l]:=-240;  yin[l]:-120; 
xin[2]:=-65;  yin[2]:=I20;  xm[3]:=-25;  yin[3]:=180;  xin[4]:=60;  yin[4]”240;  miss_ty_l  :=0;  miss_ty_2:=0;  ID:=1 11;  NEW_TRACK; 


++-H-+ 

0.030000 

3 

0 

0 

1 .000000 
1 

1000.000000 

{STRIKE  F-15E  -  ZIP  01  -  ZPOl-07,  ZPlO-12} 

iagin:=15;  limein:=0.0333;  squadin:=1072;  squad_num:=10;  speedin:=640;  altitudein:=20000;  pattem:=l ;  n:==2;  xin[l]:=93;  yin[l]:=28;  xin[2]:- 
20;  yjn[2]:=90;  miss_ty_l  :=0;  miss_ty_2:=l ;  1D:=1 1 1 ;  NEW_TRACK; 


0.090000 

3 

0 

0 

1 .000000 
1 

1000  000000 

iSTRIKEF-lGDs  -  HUSH  10-  HHlO-17,  HH20-21 } 

laiun  =16:  timcm;=0.0850;  squadin:=1077;  squad_num:=lO;  speedin:=^600;  altitudein:=32000;  pattem:=0;  n:=6;  xin[l  ]:=23;  yin[l  ]:=13;  xin[21:= 
1 10.  vin|21:=30;  xm[3):--I0;  yin[3]:=i80;  xin[4]:=-25;  yin[4]:=240;  xin[5]:=37;  yin[5]:=125;  xin[6]:=23;  yin[6]:=13;  missjy_l  :=0; 
missjy_2:=l:  10:=!  1 1 ;  NEW_TRACK; 


0  170000 
0 

1  000000 
I  • 

loooOooooo 

:ix  A  F-lOCs  -  SOL.^R  -  SRL7,  20,  21, 22} 

lagtn  =-145.  iimcin  -0.1683;  squadin:=1034;  squad_num:=4;  speedin:=600;  altitudein:=32000;  pattem;=0;  n:=6;  xin[l  ]:=23;  yin[l  ]:=13;  xin[2]:= 
UK  >in|2j  =180:  xmI3r=-25;  yin{3]:=240;  xin[4]:=-25;  yin[4]:=240;  xin[5]:=37;  yin[51:=125;  xin[6]:=23:  yin{6]:=13:  miss_ty_l  :=! ; 
ml^^  i>^2  =1.10  =111,  NFAV  TRACK; 


npiKJOo 

u 

<  I 

1  (HKMKHt 

1  ‘ 

HHK)  OIKKKM) 

Hanker  K('-10  FXXON  40  -  l:N40  41, 42  -  Alt  Change} 

tagin  =29,  iimcin  =0  1667;  squadin:=l  1 17;  altitudein:=25500,  ALT_UPDATE; 


0  170000 


0 

0 


1 .000000 
1 


A-76 


1000.000000 

{DCAF-15EII-EAGLE01-EE01-07,EE10-12!  ,  .  ,,,  -,c 

tagin:=14;  timein:=0.1667;  squadin;=1062;  squad_num:=10;  speedin:=500;  altitudein:=25000;  pattem:=0;  n;=5;  xtn[l]:=40;  yin[l]:=75; 
xin[2]:=0;  yin[2]:=60;  xin[3]:=-10;  yin[3]:=-95;  xin[4]:=-95;  yin[4]:=216;  xin[5]:=40;  yin[5]:=75;  miss_ty_l  :-0;  miss_ty_2;-l ;  1D:=1 1 1 
NEW_TRACK; 

++-I-++ 

0.190000 

3 

0 

0 

1.000000 

1 

1000.000000 

{SOJ  EF-1 1 1 A  -  Crow  01  -  CWOl  -  ID  Change} 

iagin:=7;  timein:=0.185;  squadin:=1007;  ID:=21 1;  ID_UPDATE; 


+-H-++ 

0.250000 
3  ’ 

0 

0 

1 .000000  ’ 

1 

1000.000000 

{STRIKE  F-16DS  -  HUSH  10  -  HH22-27,  HH30-33} 

tagin-143;  timein;=0.2517;  squadin:=1078;  squad_num:=10;  speedin:=600;  altitudein:=32000;  pattem:=0;  n:=5;  xm[l]:=23;  ym[l]:-13; 

xin[2]:=-l0;yin[2):=180;xin[3]:=-25;yin[3]:=240;xin[4]:=37;yin(4):=125;xin[5]:=23;yin[5]:=13;miss_ty_l:=0;miss_ty_2:=l;  1D:=111; 

NEW  TRACK: 


0.580000 
3  • 

0 

0 

1.000000 

1 

1000,000000 

{CAS  A-10  •  Warthog  51  -  WG51-56} 

iagin  =9:  iimcin:-.5767;  squadin:=1009;  squad_num:=6;  speedin:=250;  altitudein:=5000;  pattem:=l;  n:=2;  xm[l]:-35;  yin[l]:=l  15;  xin[2]:=5 
yin|21:  =  125;  miss_ty_l  :-0;  miss_ty_2:=0;  ID-1 1 1;  NEW_TRACK; 


0.610000 

3 

0 

0 

1 .000000 
1 

1000  000000 

JIX'A  MIG-29S  BAAZ  -  Hostile  -  HOOO,  HOO!  -  Course  Change} 

lamn-30.  timcm-O.Ol  33;  squadin-53;  pattem:=0;  n:==4;  xin[l  )--60;  yin[l]:=220;  xin[2]--37;  yin[2];=125;  xin[3]:=-l  10;  yin[3]:=260; 
\m\A]  =-\  10,  Mn|41:=328,  NEW_COURSE; 


1540000 

0 

0 

1  OOOOOO 

1 

lOOO.OOOOCK.) 

{STRIKE  F-1 1 1 A  -  I.AZER  30  -  LR30,  31, 32, 33, 34,  35  -  Course  Change} 

tagin:=17;  umein:=l  .54;  squadin:=1087;  patlem;=<l;  n:=3;  xin[l  ]:=-10;  yin[l  ]:=180;  xin[2]:=-95;  yin[2]:=216; 
xm[31--5;  ymI3]:=55;  NEW_COURSE; 


1 .840000 
3 


A-77 


0 

0 

1.000000 

1 

1000.000000 

{Jammers  Canberra  -  Hostile  -  H524  -New Course}  ■  r-n  • 

tagin:=136;  timein:=1.8433;  squadin;=109;  altitudein:==31000;  pattem;=0;  n:=4;  xin[i]:=-60;  yin[l]:=220;  xin[2]:-37;  yin[2]:-125;  xin[3j:— 60; 
yin[3]:=220;  xin[4]:=-27;  yin[4]:=129;  NEW_COURSE; 

-H-H-+ 

2.630000 

3 

0 

0 

1.000000 

1 

1000.000000 

(CAS  A-10  -  Warthog  51  -  WG57, 60, 61, 62} 

tagin:=144;  timein:=  2.6333;  squadin:=101 1;  squad_num:=4;  speedin:=250;  altitudein:=5000;  pattem:=l;  n:=2;  xin[l]:=35;  yin[l]:-l  15; 
xin[2]:=45;  yin[2]:=135;  miss_ty_l  :=0;  miss_ty_2:=0;  ID:=1 1 1 ;  NEW_TRACK; 


++4—f-+ 

3.010000 

3  '  ' 

0 

0 

1 .0,00000 
1 

1000.000000 

{ STRIKE  F-  111  A  -  LAZER  30  -  LR30,  31,32,  33,  34, 35  -  ID  Change } 
tagin:=l 7;  iimein:=3.0147;  squadin:=1087;  ID:=21 1 ;  1D_UPDATE; 

+++++ 

3.100000 

3 

0 

0 

1 .000000 
1 

1000.000000 

{Pakastani  Maritime  Patrol!  212  -  HOI  8} 

iagin:=6;  timem:=3.1033;  squadin:=!006;  squad_num:=! ;  speedin:=245;  altitudein:=6000;  pattem:=0;  n:=4;  xin[l]:=-l  17;  yin[l]:=120;  xin[2]:= 
65;  yinf2];=120;  xin[3]:=-57;  yin[3]”50;  xin(4]:=-l  17;  yin[4]~120;  miss_ty_l  :=0;  miss_ty_2:=0;  ID:=9999;  NEW_TRACK; 


3.170000 

3 

0 

0 

1 .000000 
1 

1000  000000 

{DC  A  F-lOCs -SOLAR  -  SRI  1,  12,  13,  14,  15,  16 -ID  Change} 
tagin  -1 1 :  umcin  =3.17;  squadin:=1032;  1D:=21 1;  1D_UPDATE; 


3.1S0000 

0 

0 

1 .000000 
I 

1000.000000 

{Jammers  Canberra  -  Hostile  -  H523,  524  -  ID  Change} 
tagin:-! 35;  limein:=3.1767;  squadin:=108;  ID:f  1 11;  ID_UPDATE; 
iagin:=l 36;  timcin:=3.l783;  squadin:=109;  ID;=1 1 1 ;  ID_UPDATE; 

’h++++ 

3.180000 


A-78 


3 

0 

0 

1.000000 

1 

1000.000000 

{Rivet  Joint  CYPHER  1 1  -  CRl  1  -  ID  Change} 

tagin:=23;  timein:=3.185;  squadin:=l  1 10;  ID:=21 1;  ID_UPDATE; 

+++++ 

3.180000 

3 

0 

0 

1.000000 

1 

1000.000000 

{ABCCC  III  EC-130ERazorback31  -RZ31  -  ID  Change) 
tagin:=26;  timein:=3.185;  squadin:=l  1 14;  ID:=21 1;  ID_UPDATE; 


3.180000 

3 

0 

0 

1 .000000 
1 

1000.000000 

jC'ompass  Call  F:C-130H  Noisy  22  -  NY22  -  ID  Change} 
tagin  =24;  timcin:=3.185;  squadin:=I  112;  ID:=21 1;  ID_UPDATE; 


3  180000 

0 

0 

l.OOOOoo 

1 

1000  000000 

ICiunshtp  .'\('-130A  Casper  51  -  CR51  -  ID  Change} 

tacin  ="27;  timcin:=03.185;  squadin:=l  1 15;  iD:=21 1;  ID_UPDATE; 


3  2fHKK«» 


(» 

1  OOOOO'I 
ItKXMXMHMM) 

;Sl  Maniimc  Surveillance  -  Cover 01  - CVOl  -  IDChange} 
tagin  '  !S.  tinicin  ~ 3  2017;  squadin:  =  1008;  ID:=211;  ID^UPDATh; 


3 

U 

0 

l.OCKMHXi 

1 

1000  000000 

{U-2R  ELINT  Snoopy  23  -  SY26  -  ID  Change} 

lagin:=28,  iimcin:=3.2017;  squadin;=l  116;  ID:=21 1;  ID_UPDATE; 


3.200000 


A-79 


3 

0 

0 

1. 000000 


1000.000000 

{Tanker  KC-10  EXXON  40  -  EN40  41 , 42  -  ID  Change} 
tagin:=29;  timein:=3.2017;  squadin:==l  1 17;  ID:=21 1;  ID_UPDATE; 


t  M  H- 

3.200000 

3 

0 

0 

1.000000 

1 

1000.000000 

{STRIKE  F-15E  -  ZIP  01  -  ZPOl -07,  ZPI 0-12  -  ID  Change} 
tagin:=15;  timein;=3.2017;  squadin:=1072;  ID:=21 1;  ID_UPDATE; 


++-H-+ 

3.220000 

3 

0 

0 

1 .000000 
I 

1000.000000 

{DCA  F-15C  Wolf  51  -  WF51 , 52,  53,  54,  55,  56,  57,  60,  61, 62,  63,  64,  65,  66  -  ID  Change} 
tagin:=10;  tjmcin:=3.2183;  squadin:=1018;  ID:=21 1;  ID_UPDATE; 


4-+++-+- 

3.250000 

3 

0 

0 

1 .000000 
1 

1000.000000 

{STRIKE  F-16DS  -  HUSH  10-  HHlO-17,  HH20-21  -  ID  Change} 
tagin:=16;  umcin:=3.2533;  squadin:=1077;  ID:=21 1 ;  ID_UPDATE; 


3.300000 

3 

0 

0 

1 .000000 
1 

1000  000000 

{CAS  A- 10  -  Warthog  51  -  WG57,  60.  61 , 62  -  ID  Change} 
iagin  =144.  timcjn  =3.3017;  squadin:=101 1 ;  ID:=21 1 ;  ID_UPDATE; 


3.340000 

3 

0 

0 

1 .000000 
1 

1000.000000 

{DCA  F-15E.  11  -  I^AGLE  01  -  EEOl-07,  EElO-12  -  IDChange} 
lagin:=14;  timcin:=3.3350,  squadin:=1062;  ID:=21 1;  ID_UPDATE; 

3.340000 

3 

0 

0 

1.000000 


A-80 


1 

1000.000000 

{DCA  F-16CS  -  SOLAR  -  SRI 7,  20,  21, 22  -  ID  Change} 
tagin:=145;  timein ’=3.335;  squadin:=1034;  ID:=211;  ID_UPDATE; 

++-H-+ 

3.380000 

3 

0 

0 

1.000000 

1 

1000.000000 

{MIG-21  Air  Defense  F-7M  1  -  Hostile  -  H050,  H054-  ID  Change} 
tagin:=52;  timein:=3.385;  squadin:=64;  ID:=1 1 1;  ID_UPDATE; 

-I-+-H-+ 

3.420000 

3 

0 

0 

1 .000000 
1 

1000.000000 

{STRIKE  F-16DS  -  HUSH  10  -  HH22-27,  HH30-33  -  ID  Change} 
iagin:=143;  limein:=3.42;  squadin:=1078;  ID:=21 1 ;  ID__UPDATE; 


3  430000 
3 
0 
0 

1 .000000 
1 

lOOOOOOOOO 

{Air  Defense  2  -  Hostile  -  H0552  -  ID  Change} 

lagin  =()S;  timein;=3.435;  squadin;=72;  ID:=I41;  ID_UPDATE; 


3.510000 

0 

i) 

I  00(MK)() 

1 

1000  l)(KM)CK) 

{Jammers  Canberra  -  Hostile  -  H524  -  ID  Change} 

lagin  =)  30.  timcin:=3.51 ;  squa4in:=109;  ID:=141 ;  ID_UPDATE; 


3  OloOon 

3 

o 

o 

1  00<>ooo 
1 

10O0  0(KK)OO 

{ I’aka.siani  Maritime  Patrol!  212  -  H018  -  ID  Change} 

lagin  =6;  tmx’in  =3.6050,  sqiiadin:=I006;  ID:=I  1 1 ;  ID_UPDATE; 


3  620000 
3 
0 
0 

1 .000000 
1 

1000  0(K  >000 

{DCA  M!Cj-29s  BAAZ  -  Hostile  -  HOOO,  HOOl  -  IDChange} 
tagin;=30;  timein:=3.615;  squadin;=53;  ID:=1 1 1;  ID_UPDATE; 


I 

3.670000 

3 

0 

0 

1.000000 

1 

1000.000000 

{PAIL72-Transport  >  ID  Change} 

tagin:=5;  timein:=3.6683;  squadin:=1005;  ID:=1 1 1;  ID__UPDATE; 

+++++ 

3.680000 

3 

0 

0 

1 .000000 
1 

1000.000000 

{Jammers  Canberra  >  Hostile  -  H524  -  ID  Change} 

tagiri:=136;  limein:=3.6767;  squadin:=109;  ID:=31 1;  ID_UPDATE; 

+++++  . 

3.770000 

3 

0 

0 

1 .000000 
1 

1000,000000 

iPakasiani  Maritime  Patrol  1212  -  HOI 8  -  IDChange} 
iagm  =(>;  iimcin:=3.77;  squadin:=1006;  ID:=121;  ID_UPDATE; 


3.840000 

0 

0 

1 ,000000 
I 

1 000  000000 

:  jammers  C'anberra  -  Hostile  -  H523  -  ID  Change} 

lagin  =135;  timcm:=3.8417;  squadin:=108;  ID:=141;  ID_UPDATE; 


3  010000 

o 

0 

1  (HKK)OO 

1 

OHMl  ()O^M)(M) 

!(  .'\S  .'\-lO  -  Warthog  51  -  \VG51-56  -  IDChange} 

lagin  nmein  =3  91 17,  squadm  =l009;  ID:=21 1 ;  ID__UPDATE; 


3  'i4i  «KH> 

3 

0 

0 

1  0(K)000 
! 

1000  000000 

I Pakasiant  Maritime  Patrol  1212 -HOIS  -  IDChange} 

lagin  =0.  limem'=3.9383;  squadin:=1006;  ID:=137;  ID^UPDATE; 


5.230000 


A-82 


3 

0 

0 

1.000000 

1 

1000.000000 

{Jammers  Canberra  -  Hostile  -  H522  -  Course  Change}  .  • 

tagin:=134;  timein:=5.2317;  squadin:=107;  pattem:=0;  n:=4;  xin[l]:=-60;  yin[l]:=220;  xm[2]:=-30;  yin[2]:=205;  xin[3]:— 60;  yin[3]:-220; 
xin[4]:=-135;  yin[4]:=245;  NEW_COURSE; 

-hH-H- 

5.320000 

3 

0 

0 

1.000000 

1 

1000.000000 

{MIG-21  Air  Defense  F-7M  1  -  Hostile  -  H054-  ID  Change} 
tagin:=52;  timein:=5.3167;  squadin:=64;  ID:=141;  ID_UPDATE; 


-+-+-M-+ 

5.400000 

3 

0 

0 

1 .000000 
1 

1000.000000 

{Jammers  Canberra  -  Hostile  -  H522  -  ID  Change } 
iagin:=134:  timein;=5.4;  squadin:-107;  ID:=1 1 1;  ID_UPDATE; 

5.570000 

3 

0 

0 

1 .000000 
1 

1000  000000 

{Jammers  Canberra  -  Hostile  -  H522  -  ID  Change} 

tagin:=l  34;  timein:=5.565;  squadin;=107;  1D:=141 ;  ID_UPDATE; 


5.670000 

3 

0 

0 

1 .000000 
1 

1000  000000 

{P.AII. 72-Transport  -  ID  Change} 

tagin  timein  =5.6667;  squadin:=1005;  ID:=21 1;  ID_UPDATE; 


5  730000 
3 
0 
0 

1 .00(K)()0 
1 

1000,000000 

{Jammers  Canberra  -  Hostile  -  H522  -  ID  Change} 

iagnr=l 34;  iimein:=5.731 7;  squadin:=107;  ID:=3 1 1 ;  ID_UPDATE; 

-h++-(-+ 

5.780000 

3 

0 


A-83 


0 

1.000000 

1 

1000.000000 

{DCA  MIG-29S  BAAZ  -  Hostile  -  HOOO,  HOOl  -  ID  Change} 
tagin:=30;  timein;=5.78;  squadin:=53;  ID:=141;  ID_UPDATE; 


6.010000 

3 

0 

0 

1.000000 

1 

1000.000000 

{Air  Defense  MIG-23  FLOGGER-B  -  H0433  -  Hostile) 

tagin:=60;  timein:=6.01 ;  squadin:=68;  squad_num:=2;  speedin:=580;  altitudein:=30000;  pattem:=l;  n:=2;  xin[l]:--l  15;  yin[l]:-21 1;  xin[2]: 
125;  yin[2]:=180;  miss_ty_l:=l;  rniss_ty_2:=l;  ID:=9999;  NEW_TRACK; 


-I-I-H-+ 

6.010000 

3 

0 

0 

1.000000 

1 

1000.000000 

{DCA  F-16CS- FRAZER  31  -  FZ31 , 32,  33, 34,  35,  36,  37,  40, 41, 42  -  Course  Change) 

tagin:=13;  timein:=6.01 ;  squadin:-1052;  pattem:=0;  n“5;  xin[l]:=l  10;  yin[l]:=30;  xin[2]:=-95;  yin[2):=216;  xin[3]:=-100;  yin[3]:=170; 
xin[4):=37;yin[4];=125;xin[5]:=35;yin[5]:=115;NEW_COURSE; 

6.010000 

3 

0 

0 

1.000000 

1 

1000,000000 

{Air  Defense  MlG-23  FLOGGER-B  -  H0433  -  Hostile  -  Course  Change) 

iagin:=60;  timem -6.012;  squadin:=68;  pattem:=0;  n:=4;  xin[l):— 100;  yin[l  ]:=170;  xin[2]:=-10;  yin[2]:==65;  xin[3]:=-100;  yin[3]:=170; 
xin(41:=-i  15;  yin|4]:=21 1;  NEW_COURSE; 

-(-+++4- 

6.180000 

3 

0 

0 

1 .000000 
.1 

1000  000000 

{Air  Defense  MlG-23  Fl.OGGER-B  -  H0433  -  Hostile  -  ID  Change) 
lagin  =60.  timcin  =6  1783,  squadin^OS;  ID:=11 1;  ID_UPDATE; 

++++•*■ 

6  280000 
3 
0 
0 

1 .000000 
1 

1000.000000 

{DCA  F-16CS  -  FRAZER  31  -  FZ31, 32,  33,  34,  35, 36,  37,  40, 41, 42  -  ID  Change) 
iagin:=13;  timcin:=6.28;  squadin:=1052;  ID:=21 1 ;  1D_UPDATE; 


+++++ 

6.340000 

3 

0 


A-84 


0 

1. 000000 
I 

1000.000000 

{Air  Defense  MIG-23  FLOGGER-B  -  H0433  -  Hostile  -  ID  Change } 
tagin:=60;  timein;=6.3433;  squadin:=68;  ID:==141;  ID_UPDATE; 

++-H-+ 

6.490000 

3 

0 

0 

1. 000000 
1 

1000.000000 

{Ground  Attack  MIG-27  BAHADUR  -  H0277  -  Hostile} 

tagin:=l  14;  timein:=6.495;  squadin:=95;  squad_num:==l;  speedin:=600;  altitudein:=29000; pattem:=0;  n:=5;  xin[l]:=-80;  yin[l]:-247;  xin[2]:= 
60;  yin[2]:=220;  xin[3]:=37;  yin[3]:=125;  xin[4]”-60;  yin[4]~220;  xin[5]:=-80;  yin[5]:=247;  missjy_l:=0;  miss_ty_2:=l;  ID:=9999; 
NEW_TRACK; 

+++++ 

6.510000 

3 

0 

0 

1 .000000 
1 

1000,000000 

1  Air  Defense  MlG-23  FLOGGER-B  -  H0433  -  Hostile  -  ID  Change} 
lagm  =00;  timcin:=6.51;  squadin:=68;  ID:=31 1;  ID_UPDATE; 


6  ()30(K)0 
3 
0 
0 

1.000000 

1 

1000  000000 

{Ground  Aiiack  MIG-27  BAHADUR  -  H0277  -  Hostile  -  ID  Change} 
lagin  =114.  omein  =6.633;  squadin:=95;  ID:=1 1 1 ;  ID_UPDATE; 


()  83O00O 
3 
0 
0 

1  OOO(MH) 

1 

lOOOOOOOfM) 

{(iround  Attack  MIG-27  BAHADUR  -  H0277  -  Hostile  -  ID  Change} 
tagin  1 14.  timem  =6  8283;  squadin;=95;  ID:=141;  ID_UPDATE; 


t.OKKMKi 

0 

o 

1  (MHHMXJ 

I 

1000(HMK)00 

{(.round  Attack  MlG-27  BAHADUR  -  H0301  -  Hostile} 

lagin  =1 1(».  iimein  =6  9116,  squadin:=96;  squad_num:=l ;  speedin:=600;  altiludein:=29000;  pattem:=0;  n:=5;  xin[!]:=-80;  yin[l];-247;  xin[2]: 
60,  >m{2|  =220,  \in[3|:=37;  yin[3]:=I25;  xin[4]:=-100;  yinl4];=170;  xin[5] --80;  yin[5]:=247;  miss_ty_l:=0;  miss_ty_2:^l;  ID;=9999; 
NIAVTRAC'K. 

7.000000 

3 

0  .  ^ 


A-85 


0 

1.000000 

I 

1000.000000 

{Ground  Attack  MIG-27  BAHADUR  -  H0277  -  Hostile  -  ID  Change} 
tagin:=l  14;  ti mein :=7. 000;  squadin:=95;  ID:=31 1 ;  ID_UPDATE; 

7.240000 

3 

0 

0 

1.000000 

1 

1000.000000 

(Ground  Attack  MIG-27  BAHADUR  -  H030I  -  Hostile  -  ID  Change} 
tagin:=l  16;  timein:=7.245;  squadin:=96;  ID:=141;  ID_UPDATE; 

7.380000 

3 

0 

0 

1. 000000 
1 

1000.000000 

{MIG-21  Air  Defense  F-7M  1  -  Hostile  -  H054-  ID  Change} 
tagin:=52;  iimein:=7.3833;  squadin”64;  ID:=31 1;  ID_UPDATE; 

++-M-+ 

7.420000 
3  ■ 

0 

0 

1 .000000 
1 

1000.000000 

{Ground  Attack  MIG-27  BAHADUR  -  H0301  -  Hostile  -  ID  Change} 
tagin;=l  16;  timcin:=7.4167;  squadin:=96;  ID:=31 1;  ID_UPDATE; 

7.430000 

3 

0 

0 

1 .000000 

1 

1000.000000 

{.^i^  Defense  2  -  Hostile  -  H0552  -  ID  Change} 

tagin:=68;  timem:-7.4333;  squadin:=72;  ID:-31 1 ;  1D_UPDATE; 


7  780000  I 

3 '  ■ 

0 

0 

1 .000000 

1 

1000.000000 

{DCA  M1G-29S  BAAZ  -  Hostile  -  HOOO,  HOOl  -  ID  Change} 
tagin  =30,  limein:-7.78;  squadin:=53;  ID:=31 1;  ID_UPDATE; 

++-(-++ 

8.110000 

3 

0 

0 

1 .000000 

I 

1000.000000 


A-86 


{Air  Defense  MIG-23  FLOGGER-B  -  H0431  -  Hostile} 

tagin;=58;  timein:=8.1 117;  squadin:=67;  squad_num:=l ;  speedin:=580;  a]titudein:=30000;  pattem:=l ;  n:— 2;  xin[l ].  1 15,  yin[l ].— 21 1 ,  xin[2]. 
125;  yin[2]:=180;  iniss_ty__l:=l;  miss_ty_2:=l;  ID:=9999;  NEW_TRACK; 


•!■++++ 

8.120000 

3 

0 

0 

1.000000 

1 

1000.000000 

{Air  Defense  MIG-23  FLOGGER-B  -  H0431  -  Hostile  -  Course  Change} 

tagin:=58;  timein:=8.12;  squadin:==67;  pattem:=0;  n:=4;  xin[l];=-60;  yin[l]:“220;  xin[2]:=30;  yin[2]:=205;  xin[3]:=-60;  yin[3].-220,  xin[4]. 
1, 15;  yin[4]:-21 1 ;  NEW_COURSE; 

+++-1-4- 

8.190000 

3 

0 

0 

1.000000 

1  .  ■ 

1000.000000 

{Air  Defense  MlG-23  FLOGGER-B  -  H0431  -  Hostile  -  ID  Change} 
tagin:=58;  timein:=8.1933;  squadin:=67;  ID:=1 1 1 ;  ID_UPDATE; 

++-»-++ 

8.360000 

3 

0 

0 

1  .oooooo 
1 

1000  000000 

I  Air  Defense  MI(i-23  FLOGGER-B  -  H043I  -  Hostile  -  ID  Change} 
iagin  =58;  iimein:=8.3583;  squadin:=67;  ID:=121;  ID_UPDATE; 


8.480000 

3 

0 

0  . 

1,000000 

1 

lOOOfMKKXK) 

{(*AS  .A-10  -  Warthog  51  -  WG57,  60,  61, 62  -  Course  Change} 

lagin  -144.  timein -8.4767;  squadin:=101 1;  pattem:=0;  n:=2;  xin[l  ];=75;  yin[l  ]:=125;  xin[2]:=35;  yin[2]:=l  15;  NEW_COURSE; 


8  510000 
3 
0 
0 

1  (MMXHKJ 
1 

lOOOO(X)OnO 

|(*.'\S  .-X-lO- W  arthog  51  -WG5 1-56  -  Course  Change} 

lagin  =0.  nmem  =^8  51;  squadin;=1009;  pattem:=0;  n;=3;  xin[l  );=5;  yin[l  ]:==125;  xin[2]:=30;  yin[2]:=160;  xin[3]:=35;  yin[3]:=l  15; 
MW  cot 'RSI  . 


8.57(KKH) 

3 

0 

0 

1 .000000 
1 

1000.000000 

{Jammers  Canberra  -  Hostile  -  H523  -  Alt  &  Spd  Change} 


A-87 


tagin:=135;  timein:==8.5717;  squadin:=108;  speedin:=500;  altitudein:=5000;  ALT_UPDATE;  SPD_UPDATE; 

I  H  M 

9.170000 

3 

0 

0 

1.000000 

1 

1000.000000 

{Air  Defense  MIG-23  FLOGGER-B  -  H0431  -  Hostile  -  ID  Change} 
tagin:=58;  timein:=9.1667;  squadin:=.67;  ID:=141;  ID_UPDATE; 


9.330000 

3 

0 

0 

1. 000000 
I 

1000.000000 

[Air  Defense  MlG-23  FLOGGER-B  -  H0431  -  Hostile  -  ID  Change} 
tagin:=58;  timein:=9.3333;  squadin:=67;  ID:=31 1;  ID_UPDATE; 


9,850000 

3 

0 

0 

1 .000000 
1 

1000.000000 

Kiround  Attack  MIG-27  BAHADUR  -  H0277  -  Hostile  -  Spd  Change} 
tagtn  =1 14;  timein;=9.855;  squadin:=95;  speedin:=550;  SPD_UPDATE; 


10  020000 
3 
0 
0 

I  000000 
I 

1000  000000 

[Jammers  Canberra  -  Hostile  -  H524  -  Spd  Change} 

laein  =136.  timein:=10.02 1 7;  squadin:=i09;  speedin:=550;  SPD^UPDATE; 


10  270000 
3 
0 
0 

1  ()(KK)O0 
! 

KKH)  (KKKHH) 

:t, round  Attack  MIG-27  BAHADUR  -  H0301  -  Hostile  -  ID  Change} 
uein  1 1(».  nmcin  --GO  2717.  squadin:=96;  speedin:=550;  SPD_UPDATE; 


l()4(»0<K)O 

3 

0 

0 

1  OOtXlOO 
1 

1000  000000 

{MlG-21  .Air  Defense  F-7M  1  -  Hostile  -  H054-  Course  Change} 

tagin  =48,  iimem:=  10.4633;  squadin;=62;  pattcm:=0;  n:=4;  xin[l  ];=-60;  yin[l  ]:=220;  xin[2]:==30;  yin[2]:=205;  xin[3]:=-60;  yin[3];=220; 
xin[41  =-48,  yinl41  =270;  Nl:\V_COURSE; 


A-88 


10.500000 

3 

0 

0  . 

1 .000000 
1 

1000.000000 

{MIG-21  Air  Defense  F-7M  1  -  Hostile  -  H050-  ID  Change} 
tagin:=48;  timein:-! 0.5033;  squadin:=62;  ID:-!  11;  ID_UPDATE; 


10.670000 

3 

0 

0 

1 .000000 
1 

1000.000000 

{MIG-21  Air  Defense  F-7M  1  -  Hostile  -  H050-  ID  Change) 
tagin;=48;  timein :==1 0.6667;  squadin:=62;  ID'.==121;  ID_UPDATE; 

+++++ 

10.900000 

3 

0 

0 

1 .000000 
1 

1000.000000 

{STRIKE  F-15E  -  ZIP  01  -  ZPOl-07,  ZPlO-12  -  Spd  Change} 
iagin:=15;  limcin:=10.905;  squadin:=1072;  speedin;=450;  SPD_UPDATE; 

10.930000 

3  ■  ■  '  . 

0 

0 

1 .000000 
1 

1000.000000 

{Ground  .Mtack  M1G-23H  FLOGGER-H  -  H0347  -  Hostile} 

ta2in:=124,  iimein:-10.9283;  squadin:=100;  squad_num:=l;  speedin:=580;  altitudein:=28500;  pattem;=0;  n:-5;  xin[l]:=-480;  yin[l]:-270; 
xinI21;=-60;  yin[2j:=220;  xin(3]:=30;  yin[3]:=205;  xin[4]:--60;  yin[4]:=220;  xin[5]:=-480;  yin[5]:=270;  miss_ty_l  :=0;  miss_ty_2:=l ;  ID:-9999; 
NT-\V  TRACK; 

11.000000 

3  ■  - . 

0 

0 

1 .000000 
1 

1000  000000 

|.MIG*21  .Air  IX'fensc  F-7M  1  -  Hostile  -  H050-  ID  Change} 
tagin  =48.  timein  =l  1 .00;  squadin:=62;  ID:=141 ;  ID_UPDATE; 


1 1  04000(1 
3 
0 
0 

1 .000000 
1 

1000.000000 

{Ground  Attack  M1G-23H  FLOGGER-H  -  H0347  -  Hostile  -  ID  Change} 
tagin:=124;  timein;=l  1 .0367;  squadin:=100;  ID:=1 1 1 ;  ID_UPDATE; 


\\  .470000 
3 


A-89 


0 

0 

1.000000 

1 

1000.000000 

{Ground  Attack  MIG-23H  FLOGGER-H  -  H0347  -  Hostile  -  ID  Change} 
tagin:=124;  timein:==l  1.4733;  squadin:=100;  ID:=121;  ID_UPDATE; 

+++++ 

11.500000 

3 

0 

0 

1.000000 

1 

1000.000000 

{MlG-21  Air  Defense  F-7M  1  -  Hostile  -  H050-  ID  Change} 
tagin~48;  timein:=l  1.50;  squadin:=62;  ID”31 1;  ID_UPDATE; 

+++++ 

12.040000 

3 

0 

0 

1 .000000 
1 

1000.000000 

{Ground  Attack  MIG-23 H  FLOGGER-H  -  H0347  -  Hostile  -  ID  Change} 
tagin:=124;  timein:=12.04;  squadin:=100;  ID:=141;  ID_UPDATE; 

++-M-+ 

13.010000 

3 

0 

0 

1 .000000 
1 

1000.000000 

{Jammers  Canberra  -  Hostile  -  H522  -  Alt  &  Spd  Change} 

tagin:=134;  timein:=l  3.0083;  squadin:=107;  altitudein:=15000;  speedin:=500;  ALT_UPDATE;  SPD_UPDATE; 

. 

13.010000 

3 

0 

0 

1. 000000 

1  .. 

1000.000000 

{MIG-21  Air  Defense  F-7M  1  -  Hostile  -  H050-  Alt  Change} 

tagin:=48;  iimein:=13.01;  squadin:=62;  altitudein:=  16000;  speedin;=500;  ALT_UPDATE;  SPD_UPDATE; 


13.330000 

3 

0 

0 

1  000000 
1 

1000  000000 

{STRIKE  F-15E  -  ZIP  01  -  ZPOl -07,  ZP 10-1 2  -  Course  Change} 

iagin:-15,  timcin:=l 3.3333;  squadin:=1072;  speedin -720;  altitudein:=30000;  pattem:=0;  n:=4;  xin[l]:=-10;  yin[l]:=180;  xin[2]:=-!00; 
yin[2):=170;  xinf3j:=37;  yin{3]:=125;  xin[4]:==93;  yin[4]:=28;  NEW^COURSE; 

+-HH-+-I- 

13.340000 

3 

0 

0 

1 .000000 


A-90 


1 

1000.000000 

{Ground  Attack  MIG-23H  FLOGGER-H  -  H0347  -  Hostile  -  Alt  &  Spd  Change }  . 

tagin -124;  timem:=I3.341 7;  squadm:=100;  altimdein:=1000;  speedin:=500;  ALT_UPDATE;  SPD_UPDATE; 

13.360000 

3 

0 

0 

1.000000 

1 

1000.000000 

{Ground  Attack  MIG-23H  FLOGGER-H  -  H0347  -  Hostile  -  ID  Change} 
tagin:=124;  timein:=13.3583;  squadin:=100;  ID:=31 1;  ID_UPDATE; 

tagin~124;  timein:=l 3.3583;  squadin:=100;  speedin:=580;  altitudein:=28500;  ALT_UPDATE;  SPD_UPDATE; 

4-++++ 

13.540000 

3 

0 

0 

1 .000000 
1 

1000.000000 

{Air  Defense  3  -  H0576  -  Hostile}  ,  . 

lagin:-78;  timein:=13.5417;  squadin:=77;  squad_num;=l;  speedin:=450;  altitudein:=5000;  pattem:=0;  n:=3;  xin[l]:=-180;  yin[l];-60;  xtn[2]:-- 
120;  yin[2]:=60;  xin[3]:=-180;  yin[3]:=60;  miss_ty_l:=l;  miss_ty_2:=l;  ID -9999;  NEW_TRACK; 


13.610000 

3 

0 

0 

1 .000000 
1 

1000  000000 

! Ground  Attack  JAGUAR  '  H0263  -  Hostile} 

iagtn  =i  12.  iimcin™  13.61 33;  squadin:=94;  squad_num:=l ;  speedin:=600;  a]titudein:=25000;  pattem:=0;  n:=5;  xin[l] —1 10;  yin[l  ]:=238; 
xinl21  =*60;  yinl21;=120;  xin[3]:=-30;  yin[3]~205;  xin|4]:=-60;  yin[41;=120;  xin[51:=-85;  yin[51:=252;  missjy_l  :=0;  miss_ty_2:-0;  ID:=9999; 
NEW  TRAc’k. 


13.670000 
3  ' 

0 

0 

1  OOOO(K) 

1 

1000  (K)OOOO 

I  An  Defense  3  •  M0576  -  Hostile  -  ID  Change} 

lagin  =^7S.  iimein  =  1 3  670;  squadin;=77;  ID:=1 1 1 ;  ID^UPDATE; 


j3S60<)0<) 

o 

(i 

1  OtXHMHl 
1 

1000  000000 

Wn  Defense  3  •  110576  -  Hostile  -  ID  Change} 

tagin  -7S.  timein  =13  8583.  squadin;=77,  ID:=121;  ID_UPDATH; 


14.04000ft 

3 

0 

0 

1 .000000 


A-91 


1 

1000.000000 

{Air  Defense  3  -  H0576  -  Hostile  -  ID  Change} 

tagin:=78;  timein:= 14.04 17;  squadin:=77;  ID.-141;  ID_UPDATE; 


-H-I-++ 

14.050000 

3 

0 

0 

1.000000 

1 

1000.000000 

{STRIKE  F-1 1 1 A  -  LAZER  30  -  LR30,  31, 32,  33, 34,  35  -  Alt  Change} 

tagin:=17;  timein:=14.05;  squadin:=1087;  speedin:=580;  altitudein:=500;  ALT_UPDATE;  SPD^UPDATE; 


14.120000 

3 

0 

0 

1 .000000 
1 

1000.000000 

{DCA  F.15E  II  -  EAGLE  01  -  EEOl-07,  EElO-12  -  Alt  Change} 
tagin;=14;  timein:=14.1 217;  squadin;=1062;  altitudein:=1000;  ALT_UPDATE; 

+++++ 

14.230000 

3 

0 

0 

1 .000000 
1 

1000.000000 

{.\ir  Defense  3  -  H0576  -  Hostile  -  ID  Change} 

tagin:=78;  timcin:=14.225;  squadin:=77;  ID:=31 1 ;  ID_UPDATE; 

-t-f4-++ 

14.670000 

3 

0 

0 

1 .000000 
1 

1000.000000 

{Ground  Attack  JAGUAR  -  H0263  -  Hostile  -  ID  Change} 
iagin:=l  12;  timein:=14.6683;  sq.uadin;=94;  ID;=31 1;  ID_UPDATE; 


15.090000 

3 

0 

0 

1 .000000 
1 

1000  0(X)000 

{Air  Defense  MICi-23  FLOGGER-B  -  H0431  -  Hostile  -  All  &  Spd  Change} 

tagin:=58;  iimein  =15.09,  squadin;=67;  speedih:=620;  altitudein:=25000;  ALT_UPDATE;  SPD_UPDATE; 


16.330000 

3 

0 

0 

1.000000 

1 

1000.000000 

{CAS  A-10  -  Warlhog  51  -  WG57,  60,  61, 62  -  All  &  Spd  Change} 


A-92 


tagin:=144;  timein:=16.3333;  squadin:=101 1;  speedm:-250;  altitudein:=5000; 


17.020000 

3 

0 

0 

1.000000 

1 

1000.000000 

{DCA  F-16CS- SOLAR- SRI  U  12,  13,  14, 15, 16  -  Alt  Change} 
tagin:=l  1 ;  timein:=17.025;  squadin:=1032;  altitudein:-500;  ALT_UPDATE; 

+++-+“1- 

17.030001 

3 

0 

0 

1.000000 

1 

1000.000000 

{Ground  Attack  MlG-27  BAHADUR  -  H0277  -  Hostile  -  Alt  Change} 
tagin:=l  14;  timein:=17.03;  squadin:=95;  altitudein:=500;  ALT_UPDATE; 


17.110001 

3 

0 

0 

1.000000 

1 

1000.000000 

{STRIKE.  F-16DS  -  HUSH  10-  HHlO-17,  HH20-21  -  ID  Change} 
tagin:=16;  umein:=17.1083;  squadin:=1077;  altitudein:=500;  ALT_UPDATE; 


17.190001 

3 

0 

0 

1 .000000 
1 

1000  000000 

{IX'A  F- IOC’s -SOLAR  -  SR17,  20,  21, 22  -  Alt  Change} 

tagin  =145,  timcin  =17.I917;  squadin:=1034;  allitudein:=500;  ALT^UPDATE; 


17  200fK)l 
3 
0 
0 

1  OOtMHH) 

1 

1(KK)0(KHHK> 

[Jammers  (  anberra  -  Hostile  -  H524  -  Alt  Change} 

tagin  =130.  timcin  =l  7.1907.  squadin;=109;  altitudein:=500;  ALT_UPDATE; 


17  27()(HH) 

3 

0 

0 

1.00(M)00 

1 

1000.000000 

{STRIKE  F-lODs  -  HUSH  10  -  HH22-27,  HH30.33  -  ID  Change} 
tagiir=143,  limcin:=  17.275;  squadin:=1078;  altitudein:=500;  ALT^UPDATE; 

17.450001 


3 

0 

0 

1.000000 

1 

1000.000000 


{Ground  Attack  MIG-27  BAHADUR  -  H0301  -  Hostile  -  Alt  Change} 
tagin:=l  16;  timein:-17.4467;  squadin:=96;  a1titudein:=500;  ALT^UPDATE; 


17.940001 

3 

0 

0 

1 .000000 
1 

1000.000000 

{Air  Defense  MIG-23  FLOGGER-B-  H0433  -  Hostile  -  Alt  Change) 
tagin:=60;  timein“l 7.9367;  squadin:=68;  aititudein:=1000;  ALT_UPDATE; 

17.940001 

3 

0 

0 

1 .000000 
1 

1000.000000 

{DCA  MIG-29S  BAAZ  -  Hostile  -  HOOO,  HOOl  -  Alt  Change} 

iagin  =30;  umein:=l  7.9367;  squadin:=53;  altitudein:=1000;  ALT_UPDATE; 


18.170000 

3 

0 

0 

i  00fK)0() 

I 

iOOOOOOOOO 

ICAS  A- 10  -  Warthog  51  -  WG51-56  -  Alt  &  Spd  Change} 

lagin  =9,  iimcin:=18.1667;  squadin:=1009;  speedin;=200;  altitudein:=500;  ALT_UPDATE;  SPD_UPDATE; 


19  OOOOOO 

3 

{} 

0 

1  0(KK.)00 
I 

ioooooooon 

I (\\S  .A*  10  .  Warthog  51  -WG51-56  -  Alt  &  Spd  Change} 

lagin  =9,  iimcin  =19  00,  squadin:=1009;  speedin;=250,  altitudein:=5000;  ALT_UPDATE;  SPD_UPDATE; 

:i  (>“()(«)() 


o 

9 

1  (MMM)OO 
KHXifKXKKH) 

!SI  Marinmc  .Surveillance  -  Cover  01  -  CVOl  -  Alt  Change} 

tagin  =8.  iimein  =21 .6667.  squadin:=1008;  allitudein:=6000;  ALT_UPDATE; 


25.080000 

0 

0 
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1.000000 

1 

1000.000000 

{MIG-21  Air  Defense  F-7M  1  -  Hostile  -  H050-  Alt  Change} 

tagin”48;  timein:=25.085;  squadin:=62;  speedin:=580;  ALT_UPDATE;  SPD_UPDATE; 


25.090000 

3 

0 

0 

1.000000 

1 

1000.000000 

{Jammers  Canberra  -  Hostile  -  H522  -  Spd  Change} 

tagin:=134;  timein:=25.0883;  squadin:=107;  speedin:=580;  SPD_UPDATE; 

25.170000 

3 

0 

0 

1 .000000 
1 

1000.000000 

{Jammers  Canberra  -  Hostile  -  H523  -  Spd  Change} 

tagin:=135;  iimcin:=25.1667;  squadin:=108;  speedin:=580;  SPD_UPDATE; 

+++'4-t- 

29.100000 
3  ■ 

0 

0 

1 .000000 
1 

1000.000000 

{STRIKE  E-1 1 1 A  -  LAZER  30  -  LR30,  3U  32, 33,  34,  35  -  Alt  Change} 
tagin  =17:  iimein;=29.1 ;  squadin:=1087;  altitudein:=32000;  ALT_UPDATE; 


29.690001 

3 

0 

0 

1 .000000 
1  . 

1000.000000 

I  DC  A  E- 1 6Cs  -  FALCON  -FN27,  30,  3 1,32} 

lacin  =12;  iimcin  =29.6950;  squadin;=1042;  squad_num:=4;  speedin:=600;  altitudein:=32000;  patlem:=0;  n:=7;  xin[l  ]:-93;  yin[l]:-28; 
xml2)  =0;  vin(2):=60;  xin[3]:=-10;  yin[3]:=180;  xin[4]:=-l  15;  yin[4]:=21 1 ;  xin[5]:=-25;  yin[5]:=240;  xin[6]:=-10;  yin[6]:=145;  xin[7]:=93 
yin[71:=28',  mis.s_iy_l:=l;  miss_ty_2-l;  ID:=9999;NEW_TRACK; 


30  530001 

3 

0 

0 

1 ,000000 
i 

1000.000000 

{DC A  F-lOCs  -  FALCON  -  FN21-26} 

tagin  =20;  iimein:=30.5283;  squadin:=1044;  squad  num:=6;  speedin;=600;  altitudein:=32000;  pattem;=0;  n:=7;  xin{l]:=93;  yin[l];=28; 
xinl2];=0;  yinI2);=60;  xin[3];=-10;  yin[3]:=180;  xin[4]:=-l  15;  yinl4];=2l  1 ;  xin[5):=-25;  yin[5]:=240;  xin[6]“-10;  yin[6]:=145;  xin[7]:=93 
yin(71:=28,  miss_ty_l;=l;  miss_ty_2:=l;  ID:=9999;  NEW_TRACK; 

31.059999 

3 

0 
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0 

1 .000000 
1 

1000.000000 

{DCA  MIG-29S  BAAZ  -  Hostile  -  HOOO,  HOOl  -  Alt  Change} 

tagin:=30;  timein:=3 1.0633;  squadin:=53;  altitudein:-35000;  ALT_UPDATE; 

+++++ 

31.559999 

3 

0 

0 

1.000000 

1 

1000.000000 

{Air  Defense  MiG-23  FLOGGER-B  -  H0433  -  Hostile  -  Alt  Change) 
tagin:=60;  timein:=3 1.5583;  squadin:=68;  altitudein:=30000;  ALT_UPDATE; 

+++++ 

31.700001 
3  . 

0 

0 

1.000000 

1 

1000.000000 

{ DCA  F- 1 6Cs  -  FALCON  -  FN27,  30,  3 1 , 32  -  ID  Change } 
tagin:=l 2;  timein:-31 .6967;  squadin:=1042;  ID:=21 1 ;  ID^UPDATE; 

+++++ 

31.799999 

3 

0 

0 

1 .000000 

1  ^  . 

1000.000000 

{DCA  F-15E  II  -  EAGLE  01  -  EEOl-07,  EElO-12  -  Alt  Change} 
tagin:=14,  iimcin:=31 .8,  squadin:=1062;  altitudein:=25000;  ALT_UPDATE; 


31.889999 

3 

0 

0 

1 .000000 
1 

1000.000000 

{Jammers  Canberra  -  Hostile  -  H523  -  Alt  Change} 

tagm -135;  timcin:=31 .8917;  squadin:=108;  altitudein:=30000;  ALT_UPDATE; 


32.310001 

3 

0 

0 

1 .000000 
1 

1000  000000 

{Ground  Attack  MlG-27  BAHADUR  -  H0277  -  Hostile  -  Alt  Change} 
lagin  -1 14.  iimcm:=32.3033;  squadin:=95;  altitudein:-29000;  ALT_UPDATE; 

+++++ 

32.470001 

3 

0 

0 

1 .000000 
1 
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1000.000000 

{Jammers  Canberra  -  Hostile  -  H524  -  Alt  &  Spd  Change } 

tagin:=136;  timein:=32.47;  squadin:=109;  speedin:=580;  altitudein:=31000;  ALT_UPDATE;  SPD_UPDATE, 

-H-H-+ 

32.529999 

3 

0 

0 

1.000000 

1 

1000.000000 

{DCA  F-16CS  -  FALCON  -  FN21-26  -  ID  Change} 
tagin:=20;  timein:=32.53;  squadin:=1044;  ID:=21 1;  ID_UPDATE; 


■H+'-H-f 

32.720001 

3 

0 

0 

1 .000000 
1 

1000.000000 

{Ground  Attack  MIG-27  BAHADUR  -  H0301  -  Hostile  -  Alt  Change} 
iagin:=l  16;  iimein:=32.72;  squadin;=96;  altitudein:=29000;  ALT_UPDATE; 


33.660000 

3 

0 

0 

1  OOOOfKJ 
1 

1000.000000 

{ Ajr  Defense  MlG-23  FLOGGER-B  -  H0431  -  Hostile  -  Alt  Change} 

iagin  =58,  iimem:=33.6633;  squadin:=67;  speedin:=580;  altitudein:=32000;  ALT_UPDATE»  SPD_UPDATE; 


33  7lKK)Ol 
3 
0 
0 

I  {)0(M)00 

!  ' 
lOOOOOOOOO 

IIX'A  I.l6('s -SOLAR- SRI  1,  12,  13,  14,  15,  1 6  -  Course  Change } 

lagin  -11,  itmein  =33.7;  squadin:=1032;  pattem:=0;  n:=3;  xin[l  ):=-25;  yin(l  ];=240;  xin[21;=37;  yin[2];=125;  xin[3]:^23;  yin[3]:=13; 

Ni  w  corRsi:. 


33S20(m>(> 

3 

o 

0 

1  OOOfMHJ 
1 

{SIRIKI  l  'l6l)s  -III  'SH  10-  HHlO-17.  HH20-21  -  Course  Change} 

lagin  -  16.  iimein  -33  8167.  squadin:=1077;  pattem:=0;  n:=3;  xinll  ]:=-25;  yin(l  ):=240;  xin[2]:=37;  yin[2]:=125;  xin[3]:=23;  yin[3]:=13; 
NFW  COI  RSI  . 


33  959990 


0 

0 

1  .OOOOOO 
1 

1000.000000 


{Ground  Attack  JAGUAR  -  H0253  -  Hostile}  ^  •  m 

tagin:=104;  timein;=33.955;  squadm:=90;  squad_nuTn:=2;  speedin:=600;  altitudein:=25000;  pattem:=0;  n:=5;  xin[l]:— 1 10;  ym[l]:-238; 
xin[2]:=-100;  yin[2]:=170;  xm[3]:=0;  yin[3]:=75;  xin[4]:=-100;  yin[4]:=170;  xin[5]:=-l  10;  yin[5]:=238;  miss_ty_l  :=0;  miss_ty_2:=0;  ID:-9999; 
NEW^TRACK; 

+++++ 

34.169998 

3 

0 

0 

1.000000 

1 

1000.000000 

{DCAF-16Cs-SOLAR-SR17.20,  21,22  -  Course  Change} 

tagin:=145;  timein:=34.1667;  squadin:=1034;  pattem:=0;  n:=3;  xin[l]:=-25;  yin[l]:=240;  xin[2]:=37;  yin[2]:-125;  xin[3]:-23;  yin[3].-13, 
NEW^COURSE; 


++-H-+ 

34.250000 

•3  '  . 

0 

0 

1 .000000 
1 

1000.000000 

{STRIKE  F-16DS  -  HUSH  10  -  HH22-27,  HH30-33  -  Course  Change} 

iagin:=!43;  timein:=34.25;  squadin:=1078;  pattem:=0;  n:-3;  xin[l]:=-25;  yin[l]:=240;  xin[2]:=37;  yin[2]:=125;  xjn[3]:=23;  yin[3]”13; 
NEW_COURSE; 

+++++ 

34.439999 

3 

0 

0 

1 .000000 
1 

1000.000000 

{DCA  F-15E  11  -  EAGLE  01  -  nE13-17,  EE20-24} 

tagin  =146:  timein:=34.435;  squadjn:=1064;  squad_num:=10,  speedin:=500;  altitudein:=25000;  pattem:=0;  n:=7;  xin[l  ]:=40;  yin[l]:-75; 
xin[2}:=0:  yin[2}:=60;  xin[3]:=-10;  yinl3];=-95;  xin[4]:=-l  15;  yin{4]  =21 1;  xin[5):=-25;  yin[5]:=240;  xin[6];=-10;  yin[6]:=145;  xin[7]:=40; 
yin[7j:=75;  niiss_ty_l  :=0;  miss_ty_2:=l;  ID:=1 1 1;  NEW_TRACK; 

++4-++ 

34.520000 

3 

0 

0 

1 .000000 
I 

loooooooori 

{ MIG-2 1  Air  IX’fcnsc  F-7M  1  -  Hostile  -  H050-  Alt  Change } 

tagin  -48:  timcin:=34.5233;  squadin:=62;  altiiudein:=29000;  ALT_UPDATE; 


34.520000 

3 

0 

0 

1 .000000 
1 

1000  000000 

{Jammers  Canberra  -  Hostile  -  H522  -  Alt  Change} 

tagin:=134,  timein;=34.5217;  squadin:=107;  altiludcin:=28500;  ALT_UPDATE: 


-K +4-1-4" 
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35.709999 

3 

0 

0 

1.000000 

1 

1000.000000 

{STRIKE  F-lllA  -LAZER30-LR36,37,40,41} 

tagin:=147;  timein:=35.7133;  squadin:=1089;  squad_num:=4;  speedin:=600;  altitudein:=32000;  pattem:=0;  n:=5;  xin[l]:=-10;  yin[l]:— 95; 
xin[2]:=-115;  yin[2]:=21 1;  xin[3]:=-25;  yin[3]:=240;  xin[4]:=-10;  yin[4]“145;  xin[5]:=-5;  yin[5]:-55;  miss_ty_l  :=0;  miss_tyj:=l;  ID;=111; 
NEW_TRACK; 

+++++ 

35.959999 

3 

0 

0 

1 .000000 
1 

1000.000000 

{Ground  Attack  JAGUAR  -  H0253  -  Hostile  -  ID  Change} 
tagin:=104;  timein:=35.9567;  squadin:=90;  ID:=1 1 1;  1D_UPDATE; 

+++++ 

36.439999 

3 

0 

0 

1 .000000 
1 

1000.000000 

{DCA  r-l5E  II  -  EAGLE  01  -  EE13-17,  EE20-24  -  ID  Change} 
tagin -146;  timein:=36.4367;  squadin:=1064;  ID:=21 1;  ID_UPDATE; 


37.720001 

3 

0 

0 

1 .000000 
1 

lOOOOOOOOO 

ISTRIKl  1-1  llA  -LA/J*:R30-LR36,37,40,41  -  ID Change} 
lagin  =14".  iimcin  =37.715;  squadin:=1089;  ID:=21 1;  ID_UPDATE; 


37.959999 

3  . .  '  ^ 

0 

0 

1  000000 
1 

l(KW)  0(MK)0() 

'Ciround  Attack  JAGl.’.AR  -  H0253  -  Hostile  -  ID  Change} 
tagin  “'104.  tinKin  =37.955.  squadin:=90,  ID:=121;  ID^UPDATE; 


38  330002 
3 
0 
0 

1  ,(KKK)(KJ 
1 

1000  000000 

{STRIKi:  I-15I-:  -  ZIP  01  -  ZP01-07,ZP10-12-Spd  Change} 

lagin  =15.  timein  =38  333;  squadin:=l072;  speedin:=500;  SPD_UPDATE; 


39.959999 


A-99 


3 

0 

0 

1.000000 

1 

1000.000000 

(Ground  Attack  JAGUAR  -  H0253  -  Hostile  -  ID  Change} 
tagin:-104;  timein:=39.955;  squadin:=90;  ID:=141;  1D_UPDATE; 

Mill 

44.000000 

3 

0 

0 

1.000000 

1 

1000.000000 

halt(); 

+++++ 

2 

601 

100.000000 

1 

10  000000 
1 

200  000000 
cic_data 

cic  hvaa  cie_fuel2  cie_arm  cie_vector 


1 

2()0 

100.000000 

I 

10  0()0(K)0 
1 

200  000000 
cxpcn 

com  time  com_h\aal  com_hvaa2  com_hostile  vector 


500 

lOOOOOOOO 

> 

KMXIOOOO 

1 

200  000000 
trace 

clock  ia>k  num  comp  l  comp_2  comp_3  com_hvaal  com_hvaa2  com_hostilc 


A- 100 


Appendix  B:  APAD  Assessor  Description  and  Code 

Introduction 


This  document  describes  the  application  programmer’s  interfaee  of  the  APAD  Bayesian  belief 
network  assessor  module. 

APAD  Assessor’s  System  Organization 

The  assessor  is  built  into  a  DLL  (Dynamic  Link  Library)  using  visual  C++.  The  Assessor  DLL 
supports  building  a  student  Bayesian  network  model  and  reasoning  within  that  model.  The 
assessor  supports  a  “C”  interface  to  LISP  and  other  languages. 

input/Output  Flies  in  the  Assessor 

File  Formats 

Rules’  Prior  File,  jhe  Rules’  prior  file  is  used  by  the  assessor  to  initialize  the  prior 
probabilities  of  the  rules  when  a  student  uses  the  ITS  for  the  first  time.  Rules’  priors  may  be 
computed  from  observations  of  students,  or  they  may  be  subjective  estimates.  All  of  the  rule 
nodes  are  assumed  to  include  two  state  values;  F  and  T.  The  Rules  prior  file’s  EBNF 
representation  is  below. 

Rule  Jile:  nde  items 
rulejtems:  nde_item 

rule_items  rule  item 

ndejtem:  rulejd  rulejiame  rulejneaning  1-  rule _prior_value  rule_pnor_value 

nde  id:  integer 

rulejiame:  identifier 

rulejneaning:  string 

prior  j'alue:  double 

After  each  rule  item,  a  new  line  is  necessary  here.  All  of  the  rules  used  in  the  assessor  are  put 
into  “Rules. txt”  as  shown  below. 

Rules  Prior  File 

1  CMoniior  Monitoring  0.5  0.5 

2  CCategorization  Categorization  0.5  0.5 

2  CROE  ROEJJse  0.5  0.5 

4  CSynibolog}’  Symbologyjknowledge  0.5  0.5 

5  Cdata  Data_gathering  0.5  0.5 

6  C system  System  jise  0.5  0.5 
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7  Cdistance  Distance_calculation _from_data_or  screen jyiew  0.5 


8 

Ccomputation  Computational _skills  0.5 

0.5 

9 

Cvisualization  Visual jcomparison  0.5 

0.5 

10 

CSA  Situational jiwareness  0.5 

0.5 

11 

Cplan  Planning  0.5  0.5 

12 

CHeuristic  Knowledge  of Jieuristic 

0.5 

0.5 

13 

CAircraftSelection  Aircraft _selection 

0.5 

0.5 

14 

CDecisionmake  Decision  making 

0.5 

0.5 

15 

CAssets  Knowledge _of_assets  0.5 

0.5 

16  CVectoring  Knowledge _of_rules  Jor_HVAA_use_given  Juel_and_armament_status 

0.5  0.5 

Old  Rules’  File,  old  rules’  file  contains  rules’  old  posterior  probabilities  of  a  student.  It  is 
created  after  the  student  solves  the  first  problem  and  is  updated  after  every  problem.  The  old 
rule  file  should  contain  the  same  rules  as  the  rules’  prior  file. 


All  of  the  rule  nodes  are  assumed  to  include  two  state  values:  F  and  T.  The  old  Rules  prior  file’s 
EBNF  is  as  same  as  Rules’  prior  file. 

APAD  Output  File,  jhe  APAD  comparison  process  output  file  is  used  as  the  evidences  file. 
The  format’s  EBNF  is  as  follows. 

Out_Put_File:  Header  Items 

Header:  String  String  String  String  String 

Items:  Item 

|Items  tern 

Item:  Int  Int  Int  bit  String  String 
bit:  Integer 

An  example  of  the  APAD  comparison  output  file  is  shown  below. 


"cie_hvaa  " 
1  I 

I  I 

3  1 

1  I 

3  I 

1  I 

3  I 

1  1 

1  1  . 

3  I 

3  I 

I  I 


APAD  Comparison  Output  File 


"cie _fuel2" 
3  1 


3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

1 


1 

1 

1 

1 

1 

1 

3 

3 

2 

2 

3 


cie_arm 
"end  260" 
"end  260" 
"end  260" 
"end  260" 
"end  260" 
"end  260" 
"end  260" 
"end  260" 
"end  260" 
"end  260" 
"end  260" 
"end  260" 


"cie_vector"  "Trigger" 
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3  11  2  "end260" 

3  0  11  "end  260" 

2  0  3  1  "end  260" 

2  0  3  1  "end260" 

APAD  Bayesian  Network,  jhe  Bayesian  belief  network  represents  APAD’s  reasoning  model. 
It  appears  in  Figure  3. 

The  Interface  Functions  of  the  APAD  Assessor 

The  following  code  represents  the  interface  functions  of  the  assessor. 

*  Name: 

*  int  Assessor_Load_Bayesian_Network(char  *fname) 

*  Function: 

*  Read  the  Bayesian  network  file  into  memory. 

*  Results: 

*  Success:  return  1 

*  Fail:  return  other 

*  Parameter: 

*  char  fiiame:  the  bayesian  network  file  name. 

*  Comment: 

*  (IjThe  file  name  should  contain  the  entire  path  necessary’  for  accessing  the  file. 

*  If  it  is  not  included  the  full  path  name,  the  default  path  is  the  current 

*  work  directory. 

*  (2)  The  following  format  are  supported: 

*  Microsoft  format  MSBN:  format  used  by  the  program  Microsoft  Bayes  Networks 

*  version  1.0. 1.7  (API).  Microsoft  MSBN32  API  library  Copyright  @  Microsoft 

*  Corp. 

*  Netica  format:  format  used  by  the  program  Netica@  version  1.06.  Netica@  is  a 

*  trade  mark  of  Norsys  Software  Corp. 

*  Ergo  format:  format  used  by  the  program  Ergo@  version  1.03.  Ergo@,  is  a  trade 

*  mark  of  Noetic  Systems  Incorporated. 

*  llugin  format :  Format  used  by  the  program  Hugint^  version  3.1.1  (API).  Hugin  is 

*  Coprighl  (a  Hugin  Expert  A/S. 

*  Dsl  format:  This  is  format  for  Genie. 

e.xtern  "C"  jleclspec(dllexport) 

int  Assessor  Load  Bayesian _Network(char  *fname); 

*  Name: 

*  void  Assessor _WriteJBayesian_Network(char  *fname) 
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*  Function: 

*  Write  the  bayesian  network  Bnet  into  a  text  file. 

*  Results: 

*  void 

*  Parameter: 

*  fname:  The  name  used  to  save  the  Bayesian  network. 

*  Comments: 

*  (l)The  fide  name  should  contain  the  entire  path  necessary  for  accessing  the  file. 

*  If  it  is  not  included  the  full  path  name,  the  default  path  is  the  current 

*  work  directory. 

*  (2)  The  following  format  are  supported: 

*  Microsoft  format  MSBN:  format  used  by  the  program  Microsoft  Bayes  Networks 

*  version  1.0. 1.7  (API).  Microsoft  MSBN32  API  library  Copyright  @  Microsoft 

*  Corp. 

*  Netica  format:  format  used  by  the  program  Netica@  version  1.06.  Netica@  is  a 

*  trade  mark  of  Norsys  Software  Corp. 

*  Ergo  format:  format  used  by  the  program  Ergo@  version  1.03.  Ergo@  is  a  trade 

*  mark  of  Noetic  Systems  Incorporated. 

*  Hugin  format :  Format  used  by  the  program  Hugin@  version  3.1.1  (API).  Hugin  is 

*  Copright  @  Hugin  Expert  A/S. 

*  Dsl  format:  This  is  format  for  Genie. 

*  This  function  writes  the  Bayesian  network  in  Ergo  format. 

*Parameters: 

*  char  *  fname:  Bayesian  network  name. 

*Results: 

*  return  1:  success 

*  return  other:  fail 
*Comments: 

*  (1)  This  function  is  used  to  initial  the  Bayesian  network, 
extern  "C"  _declspec(dllexport) 

ini  Assessor _Write_Bayesian_Network(char  *fname); 

*Name: 

*  int  Assessor _Read_Competency(char  *fname) 

^Function: 

*  This  function  reads  the  competency  rule's  old  information 

*  and  resets  the  rule  node’s  prior  by  the  post probability  value. 

*Parameters: 

*  char  *  fname:  Competency  file  name 

*Results: 

*  return  1:  success 

*  return  O.  fail 
*Comments: 

*  (1)  Make  sure  the  bnet/  rule  nodes '  name  are  fit  into  the  id's 
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*  syntax. 

*  (2)  The  rules  file  format  is  enclosed. 

*  { integer Jd  rule _id  rule jiame  prior jprobabilities  }* 

*  (3)  This  fide  should  be  called  after  Assessor_Load_Bayesian_Network(). 

extern  "C"  _declspec(dllexport) 

int  Assessor_Read_Competency(char  *fname); 

^^ti^Hii^^sti******************************************************* 

*Nmae: 

*  double  Assessor _Query_Node_Probability(  char  *  nodejd,  int  state) 

*Function: 

*  Query  node's  posterior  jprobability 
*Parameters: 

*  char  *  node  id :  node's  id 

*  int  state:  Node  evidence  state's  position 
*Results: 

*  Success:  double  value  (»  0.0  and  «  1.0). 

*  It's  the  posterior-probability  value  of  the  node. 

*  Fail:  -1.  (node's  id  is  wrong  etc.) 

^Comments:  - 

*  ( I )  Make  sure  the  bnet/  rule  nodes '  name  are  fit  into  the  id's 

*  syntax. 

e.xtern  "C"  _declspec(dllexport) 

double  Assessor _Query_Node_Probability(  char  *  node_id,  int  state); 

*Namc: 

*  int  Assesssor_Read_Evidences_RollUp(char  *file  ,char  *fname) 

^Function: 

*  Read  the  evidences  file,  and  updates  the  network,  roll  up  the  posterior 

*  probability  of  comptency>  rules  after  each  evidences  line  till  it's  finished. 
*Parameters: 

*  char  *fiile:  evidences  file  name.  The  file  is  Micro  Saint  model's  output. 

*  The  file  format  is 

*  . 

*  char  *fname:  used  to  save  the  posterior jprobability  of  the  competency  rules  after 

*  the  roll  up. 

* 

*Results: 

*  return  I:  success 

*  return  0:  fail 
^Comments: 

*  (1)  This  function  should  be  called  after  the  output  file  from  Micro  Saint 

*  model  is  generated 
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*  (&&(&cS:&3J  Here  assume  only  one  sequence  evidences 

*:!«:fc*******:|ei|esl«:(e:ic5|c5}c5jc3jc*:|c*i|cj!<5|<***s|«**********5|c************************/ 

extern  "C"  _declspec(dllexport) 

int  Assesssor_Read_Evidences_RollUp(char  *file , char  *fname); 

/************************************************♦♦************** 

*Name: 

*  int  Assessor_Close_Problem(char  *fname) 

*Function: 

*  (l)Close  one  problem  and  save  the  competency  rules'  posterior 

*  probability  into  a  file. 

*  (2)Reset  the  Bayesian  network  to  wait  for  the  next  problem. 

*  Parameters: 

*  char  *  /name:  file  name  used  to  save  the  post-probability  value 
*Results: 

*  return  1:  success 

*  return  other  value:  fail 

*Comments: 

* 

***>l<*>l<***H!>i'*=t!!|'=l'**>l<**!|t?****j|'*******=l‘*******>l‘******'l‘***************/ 

e.xtern  "C"  _declspec(dllexport) 

int  Assessor  Close _Problem(char  *fname); 

/♦♦♦^♦♦JicJlt******************************************************** 

*Name: 

*  int  Assesssor_Finished(void ) 

*F unction: 

*  Assessor  is  finished,  the  student  quit  the  system. 

*  Free  the  competency  file 
*Parameters: 

*  void 
*Resulis: 

*  return  I :  success 

*  return  other:  fail 

^Comments: 

*  This  function  should  be  called  when  the  student  quit  the  test. 

extern  "C"  _declspec(dllexport) 
int  .Assesssor _Finished(void ); 


Assessor  Application  Trace  File  Example 

This  section  demonstrates  a  “C”  trace  files  for  the  assessor.  Assessor.h  is  the  interface  file  for 
the  assessor  DLL. 


Mnclude  <stdio.h> 

#include  <stdlib.h> 

Mnclude  <string.h> 

#ifndefASSESSOR_H 
^include  "assessor.h" 
ffendif 

void  main(void) 

{ 

//Load  the  MA&D  bayesian  network 

Assessor _Load_Bayesian_Network("C:/MA_and_D/Assessor_test/MAADnet99.dsl"); 
//Read  the  rule ’s  old  posterior  probability 

Assessor _Read_Competency("C:/MA_and_D/Assessorjtest/Rules.txt"); 

//Write  the  network  into  ergo  format 
Assessor_Write_Bayesian_Network("test.erg"); 

/*Read  the  output  file  from  the  Saint  Model  and  save  the  posterior 
probabilites  of  rules  into  “test. old". 

*/ 

Assesssor_Read_Evidences_RollUp("saint _output.txt",  "test. old"); 

//Close  the  problem 

Assessor _Close_Problem(NULL); 

//Quit  the  MA&D  assessor  system 
Assesssor_Finished(); 

i 

/  . 
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Appendix  C:  Review  of  Current  Voice  Recognition  Capabilities  for 
Support  of  AWACS  Weapons  Director  Team  Research 

Background 

In  many  high  demand  workplaces,  teams  of  personnel  coordinate  their  actions  to 
accomplish  a  common  set  of  goals.  These  teams  may  be  doctors  and  nurses  in  an  emergency 
room,  a  team  of  soldiers  in  a  tank  facing  enemy  fire,  or  they  may  occur  in  many  other  high  stress 
situations  and  tense  environments.  For  our  military,  one  important  team  is  comprised  of  the 
interacting  personnel  manning  AWACS  radar  systems  and  the  pilots  who  rely  on  them  for 
guidance. 

AWACS  users  form  a  team  with  the  pilots  they  direct,  and,  as  system  users,  they 
perform  many  complex  tasks.  Many  of  these  tasks  require  extensive  verbal  communications 
with  each  other,  with  a  primary  team  comprised  of  Weapons  Directors  (WDs),  and  aircraft  pilots. 
The  WDs  interact  with  each  other  and  the  pilots  to  identify  pilot  information  support  needs.  The 
WDs  supply  the  pilots  with  tactical  guidance,  and  position  and  course  information  concerning 
other  aircraft,  and  schedule  aircraft  for  rendezvous  with  fuel  tankers.  These  communications 
consist  of  information  transmittals  necessary  for  other  personnel  to  perform  their  own  tasks. 
Thus,  for  full  coordination  of  effort  among  WDs  and  pilots,  effective  and  efficient 
communication  is  necessary. 

To  better  aid  coordinated  teams  or  groups,  such  as  WDs  and  pilots,  in  their  training  and 
performance,  extensive  research  has  been  under  way  for  many  years  to  define,  delineate,  and 
assess  individual  and  team  behaviors;  and  to  develop  automated  tools  for  supporting  teams  (see 
for  example,  Regian  &  Elliott,  1997;  or  Brannick,  et  al.,  1997).  Since  verbal  communication  is 
such  an  integral  aspect  of  the  tasks  performed  by  such  coordinated  teams,  training  and  support 
tools,  to  best  enhance  team  performance,  should  be  able  to  capture,  and  include  in  analyses,  the 
speech  acts  performed  by  team  members,  in  addition  to  capturing  other  forms  of  observable 
behavior.  The  addition  of  this  type  of  information  would  better  inform  the  identification  and 
analysis  of  task  performance  at  both  the  team  and  individual  level.  The  results  of  these  analyses 
could  then  become  the  basis  of  automated  job  aiding  or  performance  assessment  and  training 
tools. 

In  order  to  capture  speech  behavior,  several  approaches  that  can  be  taken.  First,  there  is 
the  time-consuming  process  of  tape-recording  speech,  transcribing  it  manually,  time-stamping  it, 
and  then  including  as  a  component  of  an  analysis.  Such  an  approach  is  not  conducive  to  use  in 
an  automated  system  that  would  supply  real-time  input  to  the  performer  concerning  his  or  her 
behavior.  The  ideal  method  would  be  to  capture  speech  acts  as  they  occur  during  an  on-going 
situation,  and  as  they  happen,  integrate  them  into  a  concurrent  analysis  of  performance  in  order 
to  give  real-time  feedback  to  the  producer.  This  second  method  requires  a  data  capture  system 
that  supports  automatic  speech  recognition  (ASR). 

In  the  past,  automated  voice  or  speech  recognition  capabilities  of  data  capture  systems ' 
have  been  limited  (Markowitz,  1996).  These  limitations  were  due  to  weaknesses  in  both  the 
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algorithms  used  to  recognize  specific  speakers  and  parse  speech  and  the  power  of  the  computer 
hardware  to  support  the  algorithms.  Early  systems  frequently  could  support  only  very  limited 
lexicons,  and  required  many  hours  of  training  on  a  specific  speaker’s  speech.  Even  after 
training,  the  speaker  would  have  to  talk  very  slowly  and  refrain  from  uttering  words  not 
contained  in  the  lexicon.  These  early  systems  lacked  the  power  necessary  to  allow  for  the 
integration  of  speech  acts  into  automated  analyses  of  team  or  group  task  performance. 

Recently,  however,  great  strides  have  occurred  in  ASR  technology.  Both  the  hardware  to 
support  speech  recognition  and  the  algorithms  that  are  used  in  the  process  have  increased 
significantly  in  power.  The  aim  of  this  short  report  is  to  examine  aspects  of  the  current  state  of 
ASR  research  and  available  software  to  determine  the  potential  for  the  use  of  this  technology  in 
the  development  of  team  training  and  job  support  aids,  having  a  performance  assessment 
component.  The  task  group  that  will  be  used  as  the  baseline  for  this  assessment  is  that  of 
AW  ACS  weapons  directors.  To  this  end,  this  brief  paper  will: 

•  Review  some  of  the  basic  goals  of  in  automated  speech  recognition  (ASR)  programs, 

•  Define  the  requirements  for  ASR  in  support  of  teams  such  as  WDs 

•  Present  some  of  the  more  recent  advances  in  ASR, 

•  Delineate  the  existing  problems  in  ASR,  with  indications  as  to  the  progress  toward 
their  solutions,  and 

•  Review  the  most  promising  of  reasonably  priced  Commercial  Off  The  Shelf  (COTS) 
product  to  support  the  capture  and  analysis  of  Weapons  Directors’  speech  acts. 

This  report  will  not  delve  into  the  mechanics  of  speech  recognition,  except  in  instances  in 
which  it  is  necessary  to  do  so  to  clarify  a  point.  The  field  of  research  is  large  in  both  depth  and 
breadth,  and  this  report  will  focus  only  on  what  is  pertinent  to  determining  how  such  research 
can  meet  the  goals  of  team  performance  assessment,  team  training,  and  Job  support. 


Goals  for  ASR 

According  to  Markowitz  (1996),  “Speech  is  the  ultimate,  ubiquitous  interface.”  Therefore 
it  makes  sense  as  a  design  goal,  to  develop  a  computer  system  with  which  humans  can  interact 
verbally.  That  is  one  of  the  primary  aims  of  research  in  the  area  of  ASR.  To  this  end, 
researchers  in  ASR  have  focused  on  making  the  technology  accessible  and  flexible.  For 
example,  there  arc  products  available,  already,  that  support  continuous  speech  or  specialized 
vocabularies.  There  are  systems  incorporating  ASR  that  require  no  training  on  individual 
speakers.  There  are  ASR  tool  kits  to  allow  developers  to  integrate  ASR  capabilities  into 
products.  However,  as  it  will  be  shown,  the  goal  of  speech  as  the  ubiquitous  computer  interface 
has  not  quite  been  reached. 
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Requirements  for  ASR  Systems  to  Support  Performance  of  Assessment  of  Teams 

An  ASR  component  of  an  automated  performance  assessment  or  data  capture  system  for 
teams  such  as  WDs  and  pilots  has  some  very  specific  requirements.  First,  such  a  component 
should  be  able  to  recognize  and  differentiate  multiple  voices.  In  a  situation  in  which  several 
WDs  are  passing  information  to  each  other  and  to  pilots,  the  ASR  component  will  have  to  sort 
out  the  speech  acts  of  all  participants. 

Second,  the  ASR  component  should  support  the  recognition  of  a  small,  but  specialized 
vocabulary.  It  is  possible  to  pre-train  the  ASR  component  on  the  vocabulary  prior  to  its  use, 
however. 

Third,  the  ASR  component  of  such  a  data-gathering  product  would  need  to  filter  out  a 
noisy  environment.  In  the  AWACS  environment,  there  are  extraneous  noises,  such  as  from  the 
aircraft  engines  and  from  background  chatter.  Capturing  these  noises  could  lead  to  inaccurate 
collection  of  the  target  data. 

Additionally,  the  ASR  component  should  support  continuous  speech  recognition,  because 
of  the  nature  of  types  of  verbal  tasks  performed  in  team  performance  environments.  An  ASR 
component  that  requires  the  speakers  to  slow  down  from  their  normal  rate  of  speech  could  lead 
to  an  inaccurate  picture  of  both  individual  and  team  performance.  Additionally,  such  a 
necessitated  slow-down  of  speech  could,  potentially,  lead  to  performance  errors,  which  could  be 
detrimental  in  a  real-world  situation. 

Finally,  an  ASR  product  to  be  used  as  a  component  of  an  automated  performance 
assessment  or  capturing  tool  would  need  to  be  capable  of  integration  with  other  components  of 
the  tool.  This  means  that  it  should  be  able  to  be  accessed  by  other  portion  of  the  tool  or  access 
the  other  parts,  as  specified  by  the  tool  design.  Part  of  this  integration  capability,  should  include 
the  ability  to  allow  for  time  calibration  and  time  stamping  of  the  individual  speech  acts  in 
relation  to  the  on-going  events  to  which  the  team  is  responding. 


Recent  Advances 

In  some  respects,  ASR  is  still  a  new  technology.  However,  great  strides  have  taken  place 
in  the  field  in  the  last  eight  years.  Many  of  these  advances  are  important  when  considering  the 
use  of  ASR  as  part  of  a  team  performance  assessment  or  data  capture  tool.  Some  of  these  recent 
advances  are  described  below. 

One  major  requirement  of  a  team  performance  tool  is  the  ability  to  recognize 
verbalizations  by  multiple  speakers.  Krasinski  and  Sukkar  at  AT&T  (1994)  have  developed  a 
telephone  menu  system  that  recognizes  multiple  speakers.  The  system  also  does  not  require 
training  for  recognition  of  users  speaking  words  within  the  system’s  vocabulary.  The  system 
also  “listens”  for  speech  acts  that  occur  while  an  automated  voice  is  supplying  directions  to  the 
system  users.  Finally,  the  system  is  designed  to  ignore  speech  acts  that  are  not  supported  by  its 
available  vocabulary.  It  has  a  rate  of  97.8%  successful  recognition  of  speech  acts  presented  to  it. 
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Since  the  fielding  of  this  application,  several  telephone  companies  have  fielded  similar,  very 
powerful  automated  menu  systems  that  use  ASR. 

Another  research  effort,  by  Klevens  (1997),  has  focused  on  a  combination  of  problems 
that  are  all  pertinent  to  the  team  performance  situation.  In  his  work,  he  has  developed  an  ASR 
system  to  capture  and  translate  into  text  dialogue  between  police  detectives  and  interviewees.  In 
this  type  of  event,  multiple  voices  need  to  be  captured  and  sorted.  Additionally,  there  is  no 
training  period  for  the  software  with  regard  to  the  interviewees,  nor  is  there  a  way  to  have  a 
stabilized  vocabulary  available  to  the  ASR  system.  Finally,  Klevens  was  able  to  build  in  two 
types  of  filters,  one  to  decrease  the  negative  impact  of  background  noises  to  speeeh  recognition 
and  a  second  one  to  reduce  the  distortions  to  human  voices  that  are  produced  by  sound 
transmittal  through  microphones.  The  result  of  this  effort  was  a  99.6%  accuracy  rate  for 
recognition  of  the  performed  speech  acts.  However,  in  his  conclusions,  Klevens  felt  that  more 
work  was  needed  in  the  area  of  sound  filtering,  so  that  the  accuracy  rate  of  recognition  could  be 
increased. 

As  mentioned  in  the  previous  section,  for  ASR  to  be  of  use  in  the  assessment  of  teams 
that  use  speech  acts  as  part  of  their  task  performance,  the  ability  of  the  system  to  recognize 
continuous  speech  is  imperative.  Recent  advances  in  ASR  teclmology  have  allowed  for  the 
commercialization  of  products,  such  as  products  by  Dragon  Systems,  Phillips,  and  IBM,  that  can 
recognize  up  to  160  words  per  minute,  without  pauses  between  words.  In  the  product  reviewed 
below,  one  such  ASR  system  is  examined  extensively. 

The  commercial  products  that  have  appeared  recently  on  the  market  also  promise  support 
for  extensive  and  easily  modified  vocabularies.  For  teams  such  as  WDs  and  pilots,  the  capability 
to  recognize  a  large  vocabulary,  in  itself,  is  not  important,  since  the  terms  allowed  in  the 
situation  are  limited  and  prescribed.  However,  the  capability  for  easy  voeabulary  customization 
is  important,  given  the  specificity  of  the  terminology  used  by  WDs  and  pilots. 

Finally,  to  be  useful  as  an  integrated  component  of  a  larger  automated  tool,  an  ASR 
product  needs  to  be  available  in  an  form  that  can  be  called  by  other  components  of  the  tool,  as 
needed.  There  are  several  companies  involved  in  the  development  of  ASR  tools  that  offer  such 
integratable  ASR  products  (engines)  in  the  form  of  developers’  toolkits.  One  such  package  is 
ART’S  SmARTpeak  SDK  (Software  Developer’s  Kit)  that  offers  an  Application  Program 
Interface  (API)  that  allows  ART’s  ASR  engine  to  be  integrated  with  other  applications 
developed  in  programming  languages  such  as  C  or  Visual  Basic.  A  similar  package  is  offered  by 
Voice  Control  Systems.  These  packages  seem  to  offer  a  means  to  begin  the  development  of  a 
ver>'  useful  automated  tool  that  needs  to  capture  voice  information  to  be  input  into  its  other 
functions.. 

Problems 

Although  there  have  been  extensive  advances  in  the  area  of  ASR,  many  of  these  changes 
appear  to  be  due  to  the  increase  in  power  of  the  personal  computer  systems  used  to  run  the 
applications.  With  the  increase  of  processing  speed  and  power,  has  come  the  support  for  the 
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complex  algorithms  ASR  products  use.  The  increase  in  storage  space  available  on  current  PCs 
can  support  the  extensive  vocabularies.  There  remain  several  problems  with  ASR  systems, 
however.  ASR  systems,  for  the  most  part,  still  require  close-talking  microphones,  a  single 
speaker,  and  a  relatively  quiet  background  for  optimal  performance. 

Additionally,  researchers  such  as  Dr.  Victor  Zue  of  MIT  (1985)  have  indicated  that  the 
types  of  domain-  and  user-independent  statistical  sampling  algorithms,  such  as  Hidden  Markov 
Models,  used  in  most  ASR  products  may  not  be  powerful  enough  to  allow  the  development  of 
products  that  will  overcome  current  shortcomings.  The  methodology  that  has  been  pursued  for 
over  twenty  years  is  to  develop  domain-independent  applications,  and  according  to  Dr.  Zue,  as 
early  as  1985,  this  approach  may  have  reached  its  limits  with  regard  to  extendibility.  The 
successes  in  products  that  use  such  an  approach  are  due  in  large  part  to  increases  in  power  of 
memory  availability  of  the  supporting  systems.  Some  researchers,  including  Dr.  Zue  (1995),  are 
investigating  the  development  of  ASR  systems  that  use  domain-specific  rules.  Unfortunately, 
this  new  approach  to  the  technology  is  not  moving  into  the  marketplace  very  quickly.  This 
approach  seems  to  be  potentially  useful  for  the  solution  of  the  problems  of  environmental  noise 
filtration  and  support  of  multiple  speakers.  These  issues  are  discussed  below. 

There  have  been  some  very  exciting  research  trends  using  domain-specific  rules  in  the 
area  of  environmental  noise  accommodation  for  ASR  systems.  For  example,  Nawab,  et  al., 

( 1 998)  have  been  pursuing  the  approach  of  training  ASR  systems  to  recognize  domain-specific 
noises,  so  that  the  application  can  sort  the  background  sounds  out  from  the  human  speech  acts. 
Additionally,  Klevens’  work  mentioned  above  includes  the  design  of  special  filters  that  allow  the 
ASR  system  to  “ignore”  de-humanizing  aspects  of  transmitted  sound  and  common  background 
noises. 


Similarly,  Defense  Group  Incorporated  (DGI),  has  been  focusing  specifically  on  the 
ASR-impacting  background  noise  interference  found  frequently  in  military  environments.  Their 
approach  is  to  remove  the  background  noise  via  adaptive  noise  cancellation  processes  and 
adaptive  noise  power  estimation  algorithms  coupled  with  nonlinear  noise  pre-filtering  prior  to 
subsequent  processing.  This  pre-filtering  process  is  then  combined  with  a  secondary  filtering 
process,  which  uses  a  neural  network  trained  on  the  background  noises  common  to  the 
en\  ironment.  These  noises  are  also  removed  prior  to  further  speech  processing.  More 
information  concerning  this  effort  may  be  found  on  the  Internet  at 
http:  "WWW. ca.defgrp.com/noise.html. 

Finally,  researchers  such  as  Klevens,  and  Krasinski  and  Sukkar,  above,  are  investigating 
the  issue  of  support  for  multiple  speakers.  One  approach  is  to  pre-train  the  ASR  system  to 
recognize  multiple  speakers.  That  is  the  approach  taken  by  current  off-the-shelf  products.  Other 
approaches,  implemented  in  some  break-through  packages  used  for  conference  transcription,  use 
rule-based  algorithms  rather  than  stochastically  based  approaches  to  drive  the  recognition 
process.  As  mentioned,  some  researchers,  such  as  Zue  (1995),  have  indicated  that  the  use  of 
rule-based  systems  or  neural  net  algorithms  that  better  model  the  human  auditory  process  will 
allow  future  ASR  systems  to  overcome  many  of  their  current  drawbacks.  Such  work  is  currently 
in  progress  at  Massachusetts  Institute  of  Technology’s  Laboratory  for  Computer  Science,  the 
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Automatic  Speech  Recognition  Laboratory  at  the  University  of  Illinois,  and  at  Fonix 
Corporation. 

Ability  of  COTS  ASR  Products  to  Support  Team  Performance  Assessment  and 
Job  Support 

A  search  of  the  popular  literature  in  the  computer  field  was  performed  to  select  an  off- 
the-shelf  product  for  evaluation  to  determine  its  ability  to  capture  voice  data  and  to  supply  the 
functionality  necessary  to  meet  the  requirements  of  automated  team  performance  assessment .  It 
was  decided  that  rather  than  assessing  the  performance  of  multiple  products,  only  the  product 
that  had  received  the  most  critical  recognition  and  promised  the  most  advanced  features  would 
be  examined.  It  was  felt  that  the  selection  would  best  represent  the  state-of-the-art  for 
commercially  available  products,  keeping  in  mind  that  continuing  research  efforts  promise  even 
more-greater  enhanced  products  in  the  future. 

Dragon  NaturallySpeaking  Preferred  Version  3  by  Dragon  Systems  was  selected  for 
evaluation.  This  product  was  selected  because  of  the  following  advertised  features: 

•  Support  for  continuous  speech 

•  95%  or  higher  accuracy  rate 

•  Support  for  automatically  specializing  the  recognized  vocabulary 

•  Support  for  multiple  speakers,  with  short  requirements  for  the  program  to  learn  to 
identify  the  individuals 

•  Ability  to  integrate  with  other  off-the-shelf  products  such  as  Word  for  Windows 

Each  of  these  features  was  seen  as  important  in  the  ability  of  such  a  product  to  support  the 
identification  of  verbal  task  performance  by  some  automated  assessment  tool.  The  importance  of 
each  feature  is  described  in  the  following  paragraphs. 

Support  for  Continuous  Speech_when  WDs  perform  their  tasks,  they  passing 
information  along  in  a  natural,  continuous  manner.  They  cannot  be  inconvenienced  by  an  ASR 
product  that  requires  them  to  slow  down  so  that  the  product  can  capture  their  speech.  Such  a 
slow-down  could  be  fatally  detrimental  to  their  job  performance. 

High  Accuracy  Rate — Jq  be  at  all  useful  in  performance  assessment,  either  in  support 
of  training  or  job  performance,  an  ASR  tool  must  capture  data  accurately.  Inaccurate  data  could 
lead  to  other  portions  of  the  assessment  or  training  tool  to  come  up  with  incorrect  results  and 
guidance. 

Support  for  Automatically  Specializing  the  Recognized  Vocabulary — Weapons 
Directors  and  pilots  use  a  unique  vocabulary.  It  is  not  large,  however.  Any  ASR  tool  that  is 
used  for  data  capture  must,  however,  be  able  to  recognize  the  unique  items  that  used  in  WD  and 
pilot  task  performance. 
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Support  for  Multiple  Speakers — During  a  mission  or  situation  in  which  a  team  is 
acting,  there  will  be  more  than  one  individual  speaking.  Therefore  to  be  of  any  use  in  the 
assessment  of  team  performance,  and  ASR  data  capture  tool  must  be  able  to  differentiate 
between  speakers.  Although  NaturallySpeaking  does  not  have  the  capability  to  perform  this  task 
in  that  individual  contributions  cannot  be  tagged,  it  can  recognize  multiple  speakers.  This 
indicates  that  a  future  product  could  easily  be  developed  so  that  verbal  performance  by  multiple 
speakers  within  the  same  situation  could  be  captured  and  sorted. 

Integration  Ability — NaturallySpeaking  allows  for  direct  input  into  easily  available 
word  processing  products,  such  as  Word  for  Windows.  It  is  not  a  large  step  to  go  from  input  into 
word  processing  software  into  input,  say  as  an  ASH  file,  with  utterances  time-stamped,  for 
another  application,  such  as  a  performance  assessment  tool. 


Product  Assessment 

The  selected  product  was  assessed  with  regard  to  the  following  factors: 

•  Training  requirements 

•  Multiple  speaker  recognition 

•  Ease  of  expanding  the  speech  variation  and  vocabulary 

•  Impact  of  background  noise 

Training  Requirements,  initial  training  of  the  product  was  straightforward, 
although  it  required  approximately  30  minutes  to  do  so.  The  training  consisted  of  the  author  of 
this  report  reading  a  selection  of  provided  text.  A  subsequent  test,  in  which  the  user  began  free 
dictation,  revealed  the  need  for  additional  training  to  recognize  changes  in  the  user’s  speech 
pattern  resulting  from  the  use  of  vocabulary  words  not  previously  trained. 

An  additional  training  session  of  30  minutes  was  required  to  reach  a  level  of  98% 
accuracy  for  one  user.  This  training  session  included  adding  vocabulary  from  documents  that 
were  analyzed  by  the  product  to  identify  new  words.  Also,  a  subsequent  session  in  which  the 
user  spoke  random  sentences  and  phrases  (some  of  which  would  be  similar  to  those  spoken  by 
Weapons  Directors)  and  trained  the  product  on  specific  unrecognized  phrases.  The  user  then 
read  a  portion  of  a  textbook  on  Automatic  Speech  Recognition  to  the  product  to  determine  its 
ability  to  recognize  the  words  in  the  passage.  It  was  during  this  last  test  about  the  product 
reached  the  level  of  98%  accuracy.  It  should  be  noted  that  the  product  allows  one  to  set  a  speed- 
accuracy  trade-off  level  that  is  acceptable  to  the  individual.  The  98%  level  was  reached  with  the 
product  set  with  the  speed-accuracy  trade-off  placed  at  a  middle  level. 

Multiple  Speaker  Recognition.  NaturallySpeaking  allows  for  the  development  of 
multiple  speaker  files.  The  system  is  designed  such  that  at  the  beginning  of  its  use,  one  must 
specify  the  speaker.  Then  the  software  accesses  that  user’s  speech  and  vocabulary  files  to  allow 
for  ASR  to  proceed.  Currently,  the  product  does  not  support  multiple  speakers  performing 
within  a  single  session. 
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Ease  of  Expanding  Recognition  of  Speech  Variation  and  Vocabulary.  There  are  two 
methods  for  expanding  the  speech  variation  vocabulary  for  an  individual  speaker:  1)  dictate  a 
document,  at  the  completion  of  which,  the  tool  will  ask  if  the  speech  files  should  be  updated,  and 
2)  processing  specified  documents  that  reflect  the  user’s  standard  vocabulary  or  writing  style, 
with  a  request  for  the  user  to  verbally  input  the  words  found  in  those  documents  that  have  not 
been  previously  encountered  during  training  or  use. 

Impact  of  Background  Noise,  a  brief  evaluation  of  the  product’s  ability  to  recognize  a 
user’s  speech  acts  with  a  loud  background  sound  was  performed.  Please  note  that  this  was  not  a 
rigorous  test,  but  one  to  supply  information  as  to  the  potential  utility  of  the  product  for  prototype 
performance  assessment  tools  for  team  behaviors. 

Once  the  product  was  trained  to  recognize  a  user  at  an  accuracy  rate  of  98%  over  multiple 
paragraphs  of  dictation  under  the  background  noise  conditions  found  in  a  quiet  office,  a  loud, 
continuous  background  noise  was  introduced  This  background  noise  produced  by  a  running 
large,  six-horsepower  industrial  vacuum  cleaner  placed  three  feet  from  the  user  during  dictation. 
We  felt  that  such  a  loud  noise  would  allow  some  conclusions  to  be  made  about  the  robustness  of 
the  product  with  regard  to  environmental  sound.  We  are  well  aware  that  sound  from  such  a 
vacuum  cleaner  may  not  have  all  the  properties  of  the  background  environment  noises  found  in 
an  AWACS  aircraft.  However,  we  felt  that  if  the  vacuum  cleaner  noise  interfered  with  ASR  by 
the  product,  then  it  probably  would  not  perform  acceptably  within  an  environment  with 
potentially  greater  decibels  of  sound. 

The  results  of  this  small  evaluation  were  that  the  product’s  performance  degraded  from  a 
high  of  98%  accuracy  on  recognition  of  the  contents  of  a  five  paragraph  reading  under  quiet, 
office  conditions  to  93%  on  the  same  reading  under  extremely  noisy  conditions.  However, 
training  the  product  on  individual  words  under  extremely  noisy  conditions  probably  could  boost 
the  accuracy.  Additionally,  reducing  the  speed  of  recognition  by  the  product  would  also  increase 
accuracy.  It  should  be  pointed  out  that  the  product  is  shipped  with  a  sensitive  microphone  and  an 
attached  device  (Parrot  Translator)  to  help  reduce  the  impact  of  environmental  noise.  This 
device  was  used  at  all  times  with  the  product. 

To  give  a  final  example  of  the  strength  of  the  product,  this  final  paragraph  was  dictated 
into  Word  97  for  Windows^^*  using  the  product.  Within  the  first  sentence  one  error  was  found. 
The  error  was  rapidly  corrected,  using  the  available  support  tools.  As  a  support  for  rapid 
dictation,  this  product  seems  to  be  adequate.  For  example,  this  paragraph  was  dictated  at  the 
speed  of  normal  speech.  However,  to  support  the  needs  of  Weapons  Directors  during  either 
training  or  actual  on  the  job  performance,  some  enhancements  would  be  required.  The 
concluding  section  of  this  paper  summarizes  these  requirements. 
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Conclusions 


In  this  paper,  a  brief  review  of  the  most  recent  findings  concerning  ASR  technology  as  it 
pertains  to  team  performance  data  capture  and  assessment  has  been  presented.  Additionally,  a 
rudimentary  investigation  of  capabilities  of  the  most  highly  reviewed  and  well-known  COTS 
ASR  product  was  also  performed.  From  these  two  investigations,  our  conclusion  is  that  much  of 
the  ASR  technology  is  close  to  the  state  where  it  can  be  integrated  into  team  performance  and 
communications  assessment  tools.  The  important  word  here  is  “integrated.”  There  is  ASR 
technology  available  that  will  meet  many  of  the  requirements  for  use  in  a  tool  such  as  the  one 
envisioned  in  this  paper.  However,  work  must  be  undertaken  to  bring  these  pieces  together. 

For  example,  the  state  of  the  technology  is  such  that  multiple  speaker  recognition  is 
currently  available  in  conference  transcription  software,  and  on-going  research  shows  great 
promise  for  this  technology.  Additionally,  the  work  in  environmental  noise  removal,  or 
reduction,  seems  to  be  moving  swiftly  such  that  the  capability  should  be  available  in  an  ASR  tool 
in  the  very  near  future.  ASR  technology  already  supports  continuous  speech  recognition  and 
relatively  easy  methods  for  training  voice  files,  with  profiles  being  updated  after  every  session 
with  the  user.  Finally,  there  are  programmer  tool  kits  to  allow  ASR  technology  to  be  integrated 
into  other  applications.  (Possibly,  such  an  application  kit  would  allow  for  the  integration  of  ASR 
technology  with  a  time-stamping  capability  needed  to  synchronize  other  captured  performance 
data  with  the  verbal  acts.)  These  pieces  need  to  be  brought  together  in  a  single  package  before 
one  can  begin  to  integrate  it  with  other  components  for  comprehensive  automated  performance 
assessment,  job  aiding,  or  training  tools. 
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